Cocoapuffs: Booting Fuchsia’s Zircon Kernel on a RISC-V core in Artix-7 FPGA

Zircon, the (non-POSIX, non-Linux) kernel powering Fuchsia, an open-source operating system under development at Google, now boots on my RISC-V based system-on-chip design, running on an AMD Artix-7 FPGA device. While this might be the first Fuchsia-powered device apart from the boards that Google works on, it is definitely the first piece of programmable hardware running Fuchsia. Why not Linux instead? I thought it would be novel and more amusing to see Fuchsia booting on a FPGA, vs Linux.

The resulting programmable hardware design is named “Cocoapuffs”, for reasons that are now lost to time. It is the end product of a project that I started long ago with a colleague from work. It even has a logo, straight from the pen of the creative genius that is my daughter.

Background

The complete article documenting the bringup, warts and all, is here. This design is only the tip of the iceberg of the entire toolset I developed to make it possible. I hope to tell the story about those tools one day. Fragments of that other story can be found in my bazel registry.

Full disclosure, while FPGA bringup is a personal project, Fuchsia is my day job.

It has been a long time in the making, but I am now finally able to confirm that the FPGA-based system-on-chip of my own design is capable enough to boot a real production kernel. My design loads Zircon up, and boots into userspace. At the moment, for unrelated reasons there isn’t much going on in the userspace itself, but the boot process does complete as one would expect.

Below is a video showing the boot process, slightly abridged to shorten the boring parts. In contrast to the post about the 32-bit RISC-V design, which showed the entire synthesis and programming in about 6 minutes, synthesis and P&R pass for a modern 64-bit core is serious business and as a result takes a long time. It goes on for about 1.5 hours, depending on the design particulars, and it is not very illuminating to watch, so I fast-forwarded it a bit.

If you are curious to learn more about Zircon, you can check out the official Zircon documentation, or if you prefer reading on paper, you can check out my technical report on Zircon. In either case, let me know if you have feedback.

The main component of the design is the NOEL-V RV64 soft core running on an Artix-7 (xc7a200t) FPGA. The design is a homebrew SoC I call cocoapuffs: a GRLIB noelvsys subsystem, a small SERV core that acts as the serial boot loader, DDR3, an APBUART, and the usual GRLIB plumbing, all built with Bazel rules for Vivado. The board is AX7A200B from Alinx, an exceptionally reasonably priced and well-made FPGA development board.

Zircon is not Linux. There is no earlycon, no forgiving BIOS, and the RISC-V port is comparatively young. When something goes wrong before the console comes up, you get silence. Most of this post is about turning that silence back into information. A concession to the somewhat constrained space on the xc7a200t device, the boot process is somewhat unorthodox. (One might say that xc7a200t is roomy as far as FPGA devices go. While this is mostly true, don’t forget that CPU designs are notorious for mapping poorly onto FPGA fabric, generating congestion in deeply pipelined logic and spending FPGA resouces like there’s no tomorrow.)

The rest of the article documents the final stretch to getting to a working Zircon boot.

The boot chain

The full path from power-on to userspace has a lot of links:

SERV ihex loader ──▶ OpenSBI (M-mode) ──▶ linux-riscv64-boot-shim
      ──▶ physload ──▶ physboot ──▶ Zircon kernel ──▶ userboot (userspace)

• SERV receives the firmware image over the serial line as Intel HEX and jumps to it. This is the only way onto the board — no JTAG bitstream reload between software iterations.
• OpenSBI is the M-mode supervisor firmware. It parses the devicetree, sets up the SBI runtime (timer, IPI), and drops to S-mode.
• The linux-riscv64-boot-shim is a Zircon phys executable that repackages a devicetree + ZBI into the handoff format the kernel expects.
• physload / physboot decompress and relocate the kernel, lay out physical memory, and hand off to the kernel proper.
• The Zircon kernel runs its init hooks and, finally, userboot — the first userspace process.

Every one of these stages had at least one board-specific surprise.

The bringup ladder

I like to think of a bringup like this as a ladder: each rung is a bug that hides the next one. You only ever see the lowest unsolved rung, so progress looks like a series of identical-looking hangs that each turn out to have a completely different cause.

Rung 1 — DDR3 that lies about its size

The AHB decoder maps a 1 GB window at the DDR3 base, but the physical part is only 128 MB and the window aliases every 0x0800_0000. Declaring 1 GB of RAM in the devicetree let the kernel place its data ZBI up high, where it silently wrapped back onto physical address 0 and overwrote the firmware. The fix was to tell the truth in the devicetree — one 128 MB memory@0 node — and to place the ZBI at an address that lives inside the first alias:

memory@0 {
    device_type = "memory";
    reg = <0x0 0x00000000 0x0 0x08000000>; // 128 MB, no aliasing games
};

Rung 2 — caches off, so sc.w never succeeds

GRLIB comes out of reset with the L1 caches disabled. The store-conditional (sc.w/sc.d) path in this core requires a D-cache hit, so with caches off, every store-conditional fails and any lock-based code spins forever. A three instruction boot stub before OpenSBI turns the caches on via the GRLIB cache control CSR (0x7C1):

li   t0, 0x1CF     # dsnoop | dflush | iflush | dcs=11 | ics=11
csrw 0x7C1, t0     # GRLIB CCTRL: enable I$ and D$

Rung 3 — the console goes silent because the panic path faults

This is my favorite one. The kernel would come up far enough to hit an early assert, start printing a backtrace… and then go silent. The backtrace walker in GetBacktraceCommon checked for a null frame pointer before the loop but not inside it. On this core the chain terminated with a null FP mid-walk, so the walker read from (0 - 16) + 8 == 0xFFFF…F8, faulted, re-entered the panic handler, and recursed until the console never emitted another byte. A one-line guard inside the loop restored the panic output — and the panic output is what told me about all the later rungs.

Rung 4 — a phantom memory node

An early memory@c0000000 node described the 4 KB boot BRAM as if it were 1 MB of general RAM. The PMM dutifully put page bookkeeping near the top of that window, the AHB errored the write, and you got a store access fault (scause=7) deep inside PmmNode::InitArena. Deleting the bogus node fixed it; the kernel only ever needed the real DDR3.

Rung 5 — no PLIC, no boot

The RISC-V kernel refuses to come up without an interrupt controller: interrupt_get_max_vector() returns 0 and an assert fires. NOEL-V’s GRLIB PLIC lives at 0xF800_0000; teaching the devicetree about it got us past the assert and, at last, all the way to “starting user space”.

Debugging without a console

For long stretches there was no working serial console, so the ground truth came from the RISC-V debug module. NOEL-V exposes an AHB-mapped debug module that you can reach over JTAG (BSCANE2 → an OpenOCD ahb_read config). With that I could halt the hart, read CSRs (scause, sepc, sstatus), and — crucially — dump the kernel’s debuglog ring buffer straight out of RAM. When the wire is silent, the debuglog still has the whole story:

$ openocd -f ahb_read.cfg -c 'ahb_dump 0x858400 3072'
...
starting user space
userboot: entry point                     @  0x2097aa
userboot: stack mapped                    @

I also built a small AHB transaction recorder peripheral: it passively snoops the core’s AHB bus into a BRAM ring and, on a button press, dumps the captured trace over the UART. When a hang is a bus problem rather than a control-flow problem, being able to see the last few thousand transactions is worth a great deal. (It also caused a bug of its own — more below.)

Wherever possible I reproduced things in simulation first (NVC and xsim), because a synthesis + place-and-route cycle on this design is about an hour, and you do not want an hour-long edit-compile-debug loop.

Two console gotchas worth their own section

By this point the kernel booted all the way to userspace — but only the debuglog dump over JTAG proved it. On the serial line, the boot always cut off in the same place. This turned out to be two separate problems stacked on top of each other.

The debug recorder was eating the UART

The AHB recorder I mentioned earlier is wired to steal the UART transmitter when it dumps, and the boot loader auto-triggers a dump about 30 seconds after it hands the core the CPU. That trigger fires on the first boot after programming the FPGA — right around the physboot→kernel transition — so a fresh program-then-boot never showed the kernel on the wire. (A second boot without reprogramming was clean, which is exactly the kind of Heisenbug that wastes an afternoon.) Dropping the recorder from the UART mux gave the core the serial line for the whole boot.

Userspace output needs the PLIC, not the hart

Even with the recorder gone, the console died at the last kernel init hook and userspace never appeared on serial. The cause is a nice illustration of how Zircon’s console actually works:

• Early boot writes to the UART synchronously — busy-wait on “TX ready”, push a byte. This works regardless of interrupts.
• At runtime the console is driven by the debuglog drainer thread, which does a blocking, interrupt-driven UART write: it enables the TX interrupt and then sleeps until that interrupt fires.

And the TX interrupt never fired, because the devicetree routed the UART’s interrupt to the hart-local controller (cpu0_intc) instead of the PLIC:

/* wrong: a hart-local line is never delivered as a peripheral IRQ */
interrupts-extended = <&cpu0_intc 1>;

/* right: the APBUART is PLIC source 1 in this noelvsys */
interrupts-extended = <&plic0 1>;

On RISC-V a peripheral interrupt has to arrive as an external interrupt through the PLIC. With the wrong routing, the drainer thread blocked forever the moment the TX FIFO filled so the log kept accumulating in the ring (recoverable over JTAG) but nothing after the first FIFO’s worth reached the wire.

Reparenting the UART interrupt to the PLIC was necessary but, it turned out, not sufficient: with <&plic0 1> the PLIC initializes and the kernel does switch to interrupt-driven TX (UART: IRQ driven TX: enabled), yet the APBUART’s TX interrupt on source 1 still never gets serviced — a deeper GRLIB-PLIC delivery issue I still owe a root cause. The pragmatic fix that gets the whole boot, userspace included, onto the wire is to sidestep the interrupt entirely with the kernel.debug_uart_poll=true boot argument, which polls the TX register the way early boot already does. With that, starting user space, userboot, and the final trap all print on the serial console — no JTAG required.

Where it landed

Here is the whole thing as a screencast: programming the FPGA, uploading the OpenSBI + boot-shim + Zircon image over serial (the ~24-minute upload is fast-forwarded), and the board booting:

Zircon now boots the whole chain on real hardware — OpenSBI, the boot shim, physload/physboot, the kernel’s full init, into userspace and userboot. The image I boot is a bootfs-less “eng” ZBI, so userboot correctly runs to its expected no-bootfs terminus (__builtin_trap after “no ‘/boot’ bootfs in bootstrap message”) — the same end state as the QEMU reference for that image. That is “one full proven boot”: every stage runs, in order, on silicon I can hold in my hand.