Linux boots to a shell on Vreteno

#RISC-V#Linux#FPGA#TxHDL#auto

Mainline Linux now boots to a shell on Vreteno, the RISC-V core written in TxHDL. It runs on the Alinx AX7A200B board, an Artix-7 FPGA with 1 GiB of DDR3 memory, at 100 MHz. The kernel runs in supervisor mode with virtual memory turned on. OpenSBI runs in machine mode below it, and a BusyBox shell runs in user mode above it. This post describes what the core needed for that, how the boot was brought up, and what is still missing.

What the console shows

The board loads one boot image over the network and starts it. The serial console then prints the familiar sequence:

OpenSBI v1.6
...
Linux version 6.18.55 ... (clang version 20.1.8) ...
Machine model: Alinx AX7A200B (Vreteno)
...
Run /init as init process
txhdl: userspace is up
~ #

Commands typed at the prompt answer. /proc/cpuinfo reports the core as the kernel sees it:

isa     : rv32imac_zicsr_zaamo_zalrsc_zca
mmu     : sv32
uarch   : hdlfactory,vreteno

free reports the full gigabyte of DDR3.

The plan

Before this work, Vreteno implemented RV32IMC in machine mode only. Linux for 32-bit RISC-V can run without an MMU, entirely in machine mode, and that would have been the shortest path. We chose the full configuration instead: an MMU, supervisor mode, and OpenSBI. That is the configuration ordinary Linux userspace expects.

The work was split into thirteen steps. Each step was one change with its own tests. The hardware steps were:

  • Atomics (the A extension). Load-reserved, store-conditional, and the atomic memory operations. The atomic operations compute their result from registers in the writeback stage, so the ALU’s critical path stays as it was.
  • User and supervisor modes. Trap delegation, sret, the supervisor registers, and the time and cycle counters, with the time read from the machine timer.
  • Sv32 virtual memory. Two 8-entry TLBs, one for instructions and one for data, and a page-table walker that never writes. The accessed and dirty bits are handled by page faults, which Linux supports. Each TLB looks up a registered address, so a translated bus access costs one cycle and the clock stays at 100 MHz.
  • mstatus.MPRV. OpenSBI uses it to read the kernel’s memory with the kernel’s privileges, for example to emulate a misaligned access.
  • A supervisor target on the interrupt controller, so the kernel receives device interrupts directly.
  • A SiFive-compatible serial port. Linux, OpenSBI and Zephyr all have stock drivers for it, so none of them needed a driver of ours.

The software steps were:

  • The kernel and OpenSBI, built from pinned sources with a hermetic LLVM toolchain. No GCC is involved. The kernel configuration starts from tinyconfig and adds only what this machine needs. That brought the kernel image from 35.6 MB down to 2.2 MB.
  • A userspace: musl and BusyBox, built the same way, packed into a 1.1 MB initramfs.
  • A device tree generated from the board’s own address and interrupt maps, so the tree cannot drift from the hardware.
  • A boot image that holds a small machine-mode shim, OpenSBI, the device tree, the kernel and the initramfs. The board’s network loader writes it into DDR3 and jumps to the shim.

A model of the whole machine

A kernel boot runs for hundreds of millions of instructions. The cycle-accurate simulators of the core are far too slow for that. So the bring-up used a separate model of the whole machine in software. It is built around the core’s instruction-level model, with the DDR3 memory, the timer, the interrupt controller and the serial port beside it.

The first version ran 0.65 million instructions per second. A profile showed where the time went. Every memory access walked the page tables, the bus dispatch was slow, and the model’s own library was compiled without optimisation. After a TLB in the model, a direct path to memory, devices updated only when they change, and an optimised build, it runs 16 million instructions per second. A full boot to the shell now takes about 70 seconds. It runs as a test every night.

The model found four blockers before the board saw a kernel:

  1. OpenSBI’s trap emulation needed MPRV, which the core did not have yet.
  2. The model never copied the timer’s count into the time register, so the kernel waited for a clock tick that never came.
  3. The device tree had no mmu-type, so OpenSBI marked the CPU as disabled and the kernel found no timer.
  4. The interrupt controller’s supervisor line was not connected to the core.

Each was fixed, and each fix came with a test.

On the board

The first boot on the board reached the initramfs, then stopped in the serial driver. The serial port’s receive interrupt was enabled from reset, which differs from SiFive’s reset value. A byte left over from the network loader therefore raised an interrupt request before the driver was ready for it. The model had not shown this, because it starts with no byte pending.

The boot shim now clears the serial port’s interrupts and drains the interrupt controller before it starts OpenSBI. The serial port now also resets the way SiFive’s does. The model gained a test that starts with a byte pending, to keep the case covered. The next boot on the board reached the shell.

What is missing

The setup has real limits today:

  • There is no instruction cache. Code fetched from DDR3 runs at about 15 cycles per instruction, against about 1.3 from on-chip memory. An instruction cache is the next hardware step.
  • The boot image loads slowly. The 7.5 MB image takes almost five minutes to arrive, and about half of it is padding between the parts.
  • Fast typing drops characters, because the serial port’s receive queue holds eight bytes.
  • Linux has no drivers yet for the board’s Ethernet and SD card, and it does not yet boot from flash.
  • The core’s timing margin is small. The flagship design meets 100 MHz, but only by about 0.2 ns on the core’s clock. Changes that add timing margin are in review.

Summary

Vreteno started as a test of a hardware description language. It now runs an unmodified mainline kernel with virtual memory, and the same stock drivers that run on commercial RISC-V parts. Everything from the RTL to the root filesystem is built from pinned sources with Bazel, and the whole machine boots to a shell in a nightly test. The documentation describes the core and the board in detail.