A Rocket Core on the AX7A200B, over Ethernet

#FireSim#Ethernet#FPGA#Vivado#TxHDL#RISC-V#Bazel#auto

FireSim runs simulated RISC-V systems on an FPGA. Its design with one Rocket core, Berkeley’s 64-bit RISC-V processor, now runs on an Alinx AX7A200B board. The host reaches the board over an Ethernet cable instead of PCIe, and the simulated core’s memory is the board’s DDR3. The core loaded programs, printed on its UART, read UART input, and ran a compute-bound loop at 45.7 MHz effective, on a 50 MHz FPGA clock. The full write-up is a three-page paper: FireSim over Ethernet: A Rocket Core on a Low-Cost FPGA Board (PDF).

This post continues FireSim Over Ethernet on the AX7A200B, where FireSim’s GCD example ran over the same link.

What changed since GCD

GCD needs only the simulator’s control registers. A Rocket core needs three more things:

  1. Memory. The target’s DRAM is the board’s 1 GiB of DDR3, through AMD’s MIG controller. An AXI clock converter crosses from the simulator’s 50 MHz to the controller’s 100 MHz.
  2. Streams. FireSim moves bulk data, such as the instruction traces of its TracerV bridge, through CPU-managed streams on a second, 128-bit port. These now have their own Ethernet frames, under EtherType 0x88b6, one 16-byte beat per frame.
  3. The design. FireSim’s Rocket design is large to build: elaboration and Golden Gate, FireSim’s compiler from a design to a simulator, need two Scala toolchains.

For the design, the Bazel rules now build it in continuous integration and attach the result to each release, as a 0.9 MB archive. The GitHub build of release v0.2.1 took 14 minutes. Its files are identical, byte for byte, to a build on my machine, which took 62 minutes from a cold cache. TxHDL fetches the archive by its SHA-256, so the board build needs no Scala.

The board

TxHDL’s EthShare unit reads each frame’s EtherType. Control frames go to the register master from the GCD milestone, and stream frames go to a new unit, StreamBridge. TxHDL compiles the three units together into one Verilog module. Joined wire to wire, the channels between them would form combinational loops, so the compiler puts buffers between them.

A frame can be lost, so the host repeats a request under the same tag. Neither port can safely perform a request twice: some control registers start an action, and a stream read removes data from a queue. So each master keeps the tag and the answer of its last request, and answers a repeat from them. The first request of a run is harmless if it repeats an old tag: a status read on the control path, and a no-op on the stream path.

The bitstream meets timing with 0.096 ns of setup slack, and uses 60% of the part’s LUTs and 55% of its block RAM.

The host

The host program is FireSim’s own, with its bridge drivers from Chipyard, the framework FireSim is part of. Only the platform file is new: it sends the two kinds of frame on raw sockets. A raw socket needs the cap_net_raw capability, and Linux ignores LD_LIBRARY_PATH for a binary with a file capability. So the driver links Spike, the RISC-V reference simulator that FireSim’s bridges use, statically. It is one 3.7 MB file.

The runs

Program What it checks Result
hello.riscv DDR3, UART output passed
echo.riscv UART input passed
bench.riscv speed 45.7 MHz effective
hello.riscv with TracerV the stream path failed

The first Rocket run printed this, shortened:

simif_ax7a200b_eth: device 5 on fpga-a200t-eth0, answering
FireSim fingerprint: 0x46697265
Hello from Rocket on the AX7A200B, over Ethernet.
*** PASSED *** after 68162012 cycles

bench.riscv counts a loop of 50 million iterations with the core’s own counters: 350,000,004 instructions in 600,000,056 cycles. The whole run, 687,634,397 target cycles, took about 15 s of the host’s time. FireSim reports this as an effective target clock of 45.7 MHz on the 50 MHz FPGA clock, or about 23 million instructions per second.

The GCD run measured 0.938 kHz, so the link looked slow. The difference comes from how often the host has to act. GCD needs register accesses on every step, and each access is a frame and its answer. A Rocket core runs on its own until a bridge needs the host, and this loop needs none, so the link costs almost nothing.

What does not work yet

The stream path fails on the board. With FireSim’s TracerV bridge writing a trace, hello.riscv still passed, but the trace file held only its header. The next run got no answer from the simulator, and the board needed reprogramming. The units pass their tests and their netlist simulations, and the cause on the board is still open: TxHDL issue 2007.

Also not done yet:

  • Linux on the Rocket core, with FireSim’s block-device bridge over the cable.
  • The bitstream as a prebuilt: each board build still runs Vivado.

The code is in the TxHDL repository: the board design, the platform file and the programs are under firesim/ax7a200b/, and the stream units are in lib/parts/src/remote/stream.rs.