FireSim Over Ethernet on the AX7A200B

#FireSim#Ethernet#FPGA#Vivado#TxHDL#RISC-V#auto

FireSim’s GCD example now runs on an Alinx AX7A200B board, with the host connected over a plain Ethernet cable instead of PCIe. On the board, a TxHDL unit takes register reads and writes out of raw Ethernet frames and performs them on the simulator’s control port. On the host, a 260-line platform file sends those frames from FireSim’s unchanged driver. Once the board was programmed, the test passed on its first run. The link is slow: FireSim reports an effective target clock of 0.938 kHz.

Why Ethernet

The previous post built FireSim under Bazel for the Nitefury II, a board I don’t have. FireSim’s host program reaches the FPGA over PCIe, through AMD’s XDMA core, so the board must sit in a PCIe slot. My AX7A200B has the same FPGA part, but it hangs off a server over JTAG, a serial line and Ethernet. Its Ethernet port is cabled directly to that server.

FireSim keeps all host-to-board traffic behind one C++ interface, simif_t. A platform implements two methods, a 32-bit register read and a 32-bit write on the simulator’s AXI4-Lite control port. Designs with DMA also use streams, but the GCD example needs only registers: its two bridges, peek-poke and the clock, are register blocks. So a new transport needs a new simif_t on the host and something on the board that turns the transport into AXI4-Lite accesses.

The frame

Each register access is one Ethernet frame, and each answer is another. The frames use EtherType 0x88b5, which IEEE reserves for local experiments. Since the cable runs directly between two machines, the frames need no IP address, no ARP and no DHCP.

After the 14-byte Ethernet header (destination, source, EtherType), the frame holds 13 bytes, and is padded to the 60-byte minimum:

Offset Bytes Field Ask Answer
14 1 kind 0x02 0x82
15 1 device the design’s number, 5 here the same
16 1 tag a counter the ask’s tag
17 1 flag 1 to write, 0 to read 1 on an error
18 4 address most significant byte first the same
22 4 word the data to write the data read
26 1 strobe 0xf on a write

The layout uses the same field offsets as TxHDL’s existing remote-access peripheral.

The board side

The design has four parts:

  • Clocks. One MMCM takes the board’s 200 MHz oscillator to 125 MHz for Ethernet and 50 MHz for the simulator.
  • Ethernet. TxHDL’s Gigabit MAC, behind a hand-written RGMII wrapper for the board’s JL2121 PHY. Clock-crossing FIFOs move received bytes to the 50 MHz domain and answers back to 125 MHz.
  • The bridge. HostBridge, a TxHDL unit at 50 MHz. It parses an ask, performs it as an AXI4-Lite host on the control port, and sends the answer.
  • The simulator. Golden Gate’s FPGATop, with its control port driven as single-beat AXI4 accesses.

HostBridge is two smaller units, the frame parser and the AXI4-Lite host, joined inside one lowered module. They were first two modules, wired together in Verilog. That made a combinational loop: a TxHDL sender’s valid signal depends on ready, and a receiver’s ready depends on valid. Inside one unit, the lowering puts a buffer between them.

Frames can be lost, so the host resends an ask under the same tag. The bridge keeps the tag and the answer of the last ask it performed. If an ask repeats that tag, the bridge sends the kept answer again and does not perform the access twice. Each new ask takes the next tag, so only the first ask of a run could match a tag left over from an earlier run. The driver therefore opens with a read of address 0, a status register, under a random tag, and discards the answer. If that tag matches the bridge’s kept tag, the bridge answers from what it kept. Either way nothing changes on the board, and every later tag differs from the one before it.

The bitstream meets timing with 0.534 ns of setup slack and 0.070 ns of hold slack, and with no critical warnings.

The host side

simif_ax7a200b_eth.cc implements simif_t over a Linux AF_PACKET raw socket. It sends each ask to the broadcast address on the interface named by +iface=, and waits for the answer with the same device and tag. After 20 ms without one, it sends the ask again, up to 50 times. A raw socket needs the CAP_NET_RAW capability, so the binary gets it once with setcap cap_net_raw+ep, and the driver does not run as root.

Everything else is FireSim’s own code: its main, its runtime, the GCD test and the test harness. The Bazel target links them with the new platform file and the header from Golden Gate.

The run

The first run failed on the opening read:

simif_ax7a200b_eth: no answer to a read at 0x0 after 50 tries

The bitstream had not been loaded: the programming step needs the address of the JTAG server, and it had been called without it. After programming the board, the second run passed. This is its output, shortened:

simif_ax7a200b_eth: device 5 on fpga-a200t-eth0, answering
FireSim fingerprint: 0x46697265
Commencing simulation.
* POKE GCD.io_a <- 0x40 *
* POKE GCD.io_b <- 0x30 *
* POKE GCD.io_e <- 0x1 *
...
* PEEK GCD.io_v <- 0x1 *
* PEEK GCD.io_z <- 0x10 *
* EXPECT GCD.io_z -> 0x10 ?= 0x10 : PASS
[GCD]PASS Test
*** PASSED *** after 12 cycles

Emulation Performance Summary
------------------------------
Target Cycles Emulated: 12
Effective Target Frequency: 0.938 KHz

The fingerprint 0x46697265 is “Fire” in ASCII, a constant that the simulator returns from a fixed register. Reading it back confirms that the frames reach the simulator and return. GCD(64, 48) is 16, as the test expects.

The cost

Each target cycle took about a millisecond of wall time. Every step of the target needs several register accesses, and each access is a frame out, an AXI4-Lite transaction, and a frame back, with a system call on the host for each frame. Over PCIe an access takes roughly a microsecond. For GCD the difference does not matter. For a RISC-V core booting software, which needs millions of cycles, it does.

What comes next

The next milestone is a Rocket core on the same board over the same link. It needs three more pieces:

  1. DRAM for the target, through the board’s DDR3 and AMD’s MIG controller.
  2. FireSim’s streams sent in frames, since the serial and UART bridges use them.
  3. A Golden Gate platform configuration for the AX7A200B.

The code is in the TxHDL repository: the bridge is lib/parts/src/remote/host.rs, and the board design, the platform file and the build targets are under firesim/ax7a200b/.