Automated Tests on Simulation Output, with SQLite

#EDA#VHDL#Verilog#testing#SQLite#bazel#auto

A simulator answers questions about a design, but it answers them in a waveform viewer, which is a person looking at a screen. If you want CI to answer them instead, the waveform has to become something a test can read. This is a note about turning a VCD dump into a SQLite database with go-vcd-parser, and then writing ordinary Go tests against it – go test assertions like “reset releases at 10 ns” and “this is a 200 MHz clock”, that fail a build when they stop being true.

Why not assert inside the simulation

The usual way to assert something about a simulation is to write the assertion inside it: a VHDL assert, an SVA property, an OSVVM or UVVM checker. That is the right tool when the property is about the design’s behaviour as it runs, and you should reach for it first.

Some questions are awkward there and easy afterwards, though. Post-hoc ones – “did any signal in this hierarchy stay unresolved after reset?” – are a sweep over everything the run recorded, and you often do not know you want to ask until after the run is over. Questions about the whole run are scans, not events. And a dump from a vendor tool, a colleague, or a bug report has no assertions in it and never will.

A VCD is already a log of every value change with a timestamp. That is a table. Once it is an actual table, these become tests.

From a dump to a database

# from a simulation, e.g. nvc -r --wave=dump.vcd tb
vcdcvt -in dump.vcd -format sqlite -out signals.db

That is the whole step. On an 11 MB dump with 1.2 million lines it takes about two seconds and around 19 MB of memory, because the parser streams and the loader writes batched statements with the indexes built at the end. Memory is proportional to the number of signals, not to the length of the simulation, so a long run does not need a large machine.

The result has three tables, but for writing tests you mostly do not need to know that. The dbq package is the query layer, and it talks about signals and timestamps.

A first test

The fastest way in is not to run a simulator at all. The dbt package builds a signals database in memory from literal time/value pairs, which is how dbq’s own tests are written – see dbq/pkg_test.go for the maintained examples. This is the right shape for testing your checking logic: fast, hermetic, and no waveform in sight.

dbx, _ := db.OpenDB(ctx, dbt.NewMemDB())
i := dbt.New(dbx, ctx)
i.Signal("//clk", vcd.VarKindLogic, 1).
    //
    // //clk   ________/~~~~~~~~~~...
    //         ^0      ^100
    TimeValues(dbt.TimeValue{Time: 0, Value: "0"},
        dbt.TimeValue{Time: 100, Value: "Z"})

q := dbq.New(dbx)
ts := q.Signal("//clk").FindFirst("Z")   // ts.T() == 100

The vocabulary is small and composes: FindFirst(value), FindAfter(ts, value), FindBefore(ts, value), EqAt(ts, value), PrevChange(ts) and NextChange(ts). Two value lookups differ in a way worth memorising – ValueAt(ts) gives the value before a transition exactly at ts, while ValueAtP(ts) includes it. Getting that backwards is the usual off-by-one-edge bug.

Every lookup returns a Timestamp that may be absent or carry an error, so checking is explicit:

ts := clk.FindFirst("1")
if err := ts.IsOk(); err != nil {
    t.Fatalf("clock never rises: %v", err)
}

dbq.FindFirst composes several lookups into “the first moment at which all of these hold at once”, which is the shape most protocol checks take:

r := dbq.FindFirst(
    func(ts *dbq.Timestamp) *dbq.Timestamp { return s1.FindAfter(ts, "1") },
    func(ts *dbq.Timestamp) *dbq.Timestamp { return s2.EqAt(ts, "2") },
    func(ts *dbq.Timestamp) *dbq.Timestamp { return s3.EqAt(ts, "3") },
)

Testing against a real simulation

The same queries run against a database built from an actual dump. Under Bazel the conversion is a build step, so the database is a build artifact rebuilt whenever the dump changes, and the test that reads it caches when neither has:

load("//:macros.bzl", "vcd_index", "vcd_go_test")

vcd_index(
    name = "tb_example",
    vcd_target = "tb.vcd",
)

vcd_go_test(
    name = "dbq_test",
    srcs = ["pkg_test.go"],
    embed = [":dbq"],
    vcd_file = "//vcd/files/samples:tb_example",
)

vcd_index runs vcdcvt in a genrule. vcd_go_test wraps go_test, passes the database in as data, and hands the test its path with --test-db-name. Inside the test, dbq.GetTestDB() opens it:

func TestResetReleasesBeforeTraffic(t *testing.T) {
    dbx, _, err := dbq.GetTestDB()
    if err != nil {
        t.Fatalf("no test database: %v", err)
    }
    q := dbq.New(dbx)

    release := q.Signal("//wb_uart_tb/reset").FindFirst("0")
    if err := release.IsOk(); err != nil {
        t.Fatalf("reset never released: %v", err)
    }
    if got, want := release.D(), 10*time.Nanosecond; got != want {
        t.Errorf("reset released at %v, want %v", got, want)
    }
    // reset_n is the inverse, and must already be high here.
    resetN := q.Signal("//wb_uart_tb/clkgen/reset_n")
    if v := resetN.ValueAtP(release); v.V() != "1" {
        t.Errorf("reset_n is %q when reset releases, want \"1\"", v.V())
    }
}

Nothing in that test knows a simulator exists. It reads a file the build produced, which means it runs in the same sandbox as every other test, in parallel with them, and its failure message names the signal.

Signal names are the full hierarchical path: //wb_uart_tb/clkgen/reset_n. Run vcdcvt with -signals signals.csv to get a listing of every name, type and width, which is how you find out what a hierarchy is called.

Timing assertions, and the unit time is counted in

Timestamp.D() returns a time.Duration, and three helpers build on it: Diff(a, b) for the gap between two timestamps, IsDurationApprox(a, b, d) for “these are d apart”, and IsClock(from, sig, freq), which walks a rising edge, the falling edge after it and the next rising edge, and checks both half periods:

clk := q.Signal("//wb_uart_tb/clk")
if err := dbq.IsClock(&dbq.TimestampZero, clk, 200e6); err != nil {
    t.Errorf("clk is not a 200 MHz clock: %v", err)
}

The thing to understand here is that a raw timestamp is not nanoseconds. It is an integer count of whatever unit the file’s own $timescale declares, and simulators disagree: nvc will happily write 1fs, Vivado’s xsim often 1ps. A count of 1000000 is one nanosecond in the first case and one microsecond in the second.

The conversion records the timescale and dbq reads it, so the helpers above hold whatever the file used. It is worth knowing that they did not always: D() used to assume one picosecond per tick unconditionally, which is exact for a 1ps dump and a factor of a thousand out for a 1fs one – reset “released at 10 µs”, and IsClock could not see a 200 MHz clock at all. Writing this article is what turned that up. If you are on an older version, check before trusting an absolute duration.

One limit remains, and it is a property of time.Duration rather than a bug: it counts whole nanoseconds. Half a nanosecond of skew in a 1fs dump is not representable, and D() reports zero for it. Fs() returns femtoseconds and is exact for every timescale a VCD can declare, so sub-nanosecond work goes through that:

skew := later.Fs() - sooner.Fs()   // femtoseconds, exact

Ratios were always safe either way, since the scale factor cancels.

When a test fails

A failing assertion gives you a signal name and a timestamp, which is exactly what you need to go and look at the waveform. The companion tool turns a window of the database back into a picture:

sqlite2drawtiming -in signals.db -min-time 0 -max-time 40000000 \
    -ndots 2500000 \
    -signal '//wb_uart_tb/clk=>clk' \
    -signal '//wb_uart_tb/reset=>reset' > timing.dt
drawtiming -o timing.png timing.dt

A reasonable loop to end up in: assert in CI, and when CI complains, render the exact window it complained about.

If you would rather write SQL

It is a plain SQLite file with no server, so nothing stops you querying it directly, and a few questions are much easier that way – particularly sweeps across every signal, or across the whole run, which the Go API does not try to express. Three tables:

Signals(Name, Type, Code, Size)     -- Type 17 is logic, 18 is string
Svalues(Id, Timestamp, Code, Value) -- Code joins to Signals.Code
Meta(Key, Value)                    -- generator, timescale, timescale_seconds

“Did anything, anywhere, fail to resolve after reset?” is one query, where as a Go test it would mean enumerating the hierarchy by hand:

WITH settled AS (
  SELECT s.Name AS name, v.Value AS value,
         ROW_NUMBER() OVER (
           PARTITION BY v.Code ORDER BY v.Timestamp DESC) AS rn
  FROM Svalues v JOIN Signals s ON s.Code = v.Code
  WHERE v.Timestamp <= 20000000   -- 20 ns, after reset releases
    AND s.Type <> 18)             -- 18 is $var string
SELECT name, value FROM settled
WHERE rn = 1 AND value GLOB '*[XxUuZzWw-]*'
ORDER BY name;

ROW_NUMBER() ... PARTITION BY v.Code is the idiom for “the last value each signal took at or before time T”. On the UART testbench this reports nine genuinely undriven signals.

Two traps are hiding in that one predicate. SQLite’s LIKE is case-insensitive for ASCII, so the obvious value LIKE '%X%' also matches the x in rx_get_start_bit and reports a healthy state machine as broken. GLOB is case-sensitive and fixes that, then breaks the other way: the same dump writes vector metavalues as uppercase X but a scalar U as lowercase u, so GLOB '*U*' silently misses it. Hence the explicit character class, and the Type filter to keep the string-valued enums out of it entirely.

Timestamps in SQL are raw ticks, so scale them yourself – Timestamp * (SELECT CAST(Value AS REAL) FROM Meta WHERE Key = 'timescale_seconds') gives seconds. Queries like these stay fast because the converter indexes Svalues(Code, Timestamp, Value), which covers the per-signal lookups outright.

Worth knowing

  • The database is one file with no server, so it is an ordinary build artifact: cacheable, copyable, attachable to a bug report.
  • The schema is small enough that a different writer – a .wdb converter, say – can produce the same tables, and then one test suite works across both.
  • Assertions inside the simulation still catch things earlier and with better context. This is a complement to them, not a replacement.