manuel313/famaf_cva6 repository overview

⁠The CVA6 container

The real-hardware side of the FaMAF CVA6 Project: the frozen CORE-V CVA6 fork, a built RISC-V toolchain, Verilator, Spike, the run drivers and the CVA6Flow viewer, in one image. Everything lives under /CVA6, which is the working directory of every shell the image starts.

⁠Quick start

docker pull manuel313/famaf_cva6
docker run -dit --name CVA6 -p 127.0.0.1:8001:8000 --shm-size 2g manuel313/famaf_cva6 bash
docker exec -it CVA6 bash

The port is published on the host's loopback alone, so a served viewer is not offered to the rest of the network, and --shm-size 2g is the shared memory a Verilator build needs. Then, inside the container:

python3 scripts/run_CVA6.py benchmarks/config/store_fwd.S       # builds the Verilator model first, about 2 minutes
python3 CVA6Flow/CVA6Flow_tracer.py results/run/store_fwd.vcd -o results/run/store_fwd.json
python3 scripts/make_CVA6Flow_sample.py results/run/store_fwd.json -i 0
python3 scripts/serve_CVA6Flow.py    # then open http://localhost:8001/CVA6Flow/CVA6Flow.html

The page opens a JSON from the host's disk, or a sample the server lists, which is what the third line makes of this one, so it is offered under Load Sample as store_fwd. Load Sample works straight away too: the image carries a sample of each of its nine programs, the whole run for the five that fit the 500,000 records the page renders at once, made while it was built, and the repository's sample, cut from the start of Main Code, for the other four. With the repository cloned, python3 scripts/make_containers.py CVA6 --pull does the same with limits sized to your machine, and scripts/docker_sync.py moves files between the checkout and the container.

⁠Tags

Pull v2.0.0 and put that tag in the commands above to stay on 2.0.0, the release whose results the repository reports.

⁠Its counterpart

manuel313/famaf_gem5⁠ runs the same benchmarks on gem5's MinorCPU, matched to this core. The project, its documentation and the recipe for this image are at https://github.com/FaMAF-CVA6-Project/CVA6⁠, and FlowCompare, the comparison viewer that puts the two side by side, is live at https://famaf-cva6-project.github.io/CVA6/⁠.

Everything below describes what is inside the container, so every path is a container path.

⁠What is in here

The target built here is cv64a6_imafdc_sv39_hpdcache_wb, a 64-bit CVA6 with the HPDcache writeback data cache, a 16 KiB 4-way instruction cache and a 32 KiB 8-way data cache.

⁠Run one benchmark

python3 scripts/run_CVA6.py benchmarks/config/store_fwd.S --no-vcd

A bare name is enough, store_fwd.S: programs are looked up in benchmarks/config/ and then benchmarks/viewer/, and when both hold one, as they do daxpy.S, the first is run and the other named, so give the path to run the viewer's.

The first run compiles the Verilated model into work-ver/, which took about a minute and a half on a 16-thread machine. Every run after that can reuse it with --keep-build, as long as the target, the VCD setting, the configuration package and the testbench changes are all as they were when it was built. A sweep leaves it built for its last configuration. One run goes at a time: a second run_CVA6.py started while one runs is refused, since both would build into work-ver/.

What it prints is a metrics table with two figures per metric: OFFICIAL is what the hardware counters read, and NET subtracts the fixed instrumentation overhead of the suite the program belongs to, which is the number the gem5 side is compared against.

Output lands under results/: the files worth keeping in results/run/, and the simulator's own output folder moved to results/verif/out_<date>/ when the run survives to the end.

⁠The other drivers

Every one of them answers --help, and all but the server take -n, or --dry-run, to say what they would do and do nothing: the cleaner lists what it would delete, the batch and both sweeps print the plan, the batch JSON converter the VCDs it would convert, measure_CVA6_overhead.py its runs, and patch_CVA6_testbench.py rehearses the change on a copy of the files it touches.

⁠Use the viewer

python3 scripts/serve_CVA6Flow.py

It listens on port 8000 inside the container, which make_containers.py publishes as 8001 on the host's loopback, so open http://localhost:8001/CVA6Flow/CVA6Flow.html there. The gem5 container takes 8000, so both can serve at the same time.

The viewer loads the JSON the tracer makes from a VCD:

python3 CVA6Flow/CVA6Flow_tracer.py results/run/daxpy.vcd -o results/run/daxpy.json

The page opens a JSON from the host's disk, dropped onto it or chosen with Load JSON, so either copy the JSON out to the host, with docker cp or the fork's scripts/docker_sync.py pull, or make a sample of it inside the container, which the served page then offers by name:

python3 scripts/make_CVA6Flow_sample.py results/run/daxpy.json -i 0

-i 0 keeps every record, and the script refuses a JSON of more than the 500,000 records the page renders at once. Without -i it keeps the first 3,000. The sample lands in CVA6Flow/tests/ as daxpy.sample.js, listed in CVA6Flow/tests/samples.js, and the page scripts/serve_CVA6Flow.py serves offers it in its sample picker.

The tracer finds the disassembly listing on its own when it sits beside the VCD under the same name, and --disasm-list names it otherwise. With --strict the tracer exits with 3 on a degraded VCD: one missing the signals of a mechanism the tracer follows, so that mechanism could not be observed at all, or one that was cut, holds no value changes, no rising clock edge, no instruction or no commit, starts part way through the run, as a windowed dump does, ends at a timestamp that disagrees with its cycle count, or ends before the program does, which the listing beside it shows. metadata.degraded always names each one, whether or not --strict is passed. The viewer refuses a JSON whose metadata.schema_version is not 4 and asks for it to be regenerated, rather than drawing silent nulls.

CVA6Flow/tests/ already holds a full sample of every program in benchmarks/viewer/ whose run fits the 500,000 records the page renders at once, every record of it, made while this image was built, so Load Sample works before anything has been run here. The four that run past that limit, commit_ilp_test, data_cache_stress_test, icache_footprint_test and store_miss_test, keep the repository's sample, cut from the start of Main Code. The page offers only what tests/samples.js lists that its tracer wrote at schema 4, so a sample made later appears too once its .sample.js and the manifest are in CVA6Flow/tests/. On the host, python3 scripts/docker_sync.py push CVA6 replaces the folder with the checkout's own.

⁠Testbench changes

verilator_changes/ holds two changes to the testbench, both out as the image is built, each a diff against the upstream files that scripts/patch_CVA6_testbench.py puts in and takes out, alone or together:

• vcd_window/ bounds the waveform dump to a window of clock cycles, since an unwindowed VCD of a long benchmark reaches tens of gigabytes. The window, in clock cycles with both ends included, whose VCD timestamps are twice the cycle numbers, is exported before the run as vcd_start and vcd_end, or given to the model itself as +dump-start= and +dump-end=.
• ddr3_memory/ replaces the single-transaction axi2mem with a DDR3-1600 model of gem5's DDR3_1600_8x8 and its MemCtrl, printing its statistics into the run's log, results/verif/out_<date>/veri-testharness_sim/<test>.*.log.iss once the run has ended, and under verif/sim/out_<date>/ while it runs or when it fails. ddr3_memory/tb/ is a standalone testbench for the model, make there with tools/verilator/bin on the path.

python3 scripts/patch_CVA6_testbench.py apply --vcd-window --ddr3
export vcd_start=100000 vcd_end=200000
python3 scripts/run_CVA6.py benchmarks/viewer/daxpy.S
python3 scripts/patch_CVA6_testbench.py revert

Either change needs the model rebuilt, which the first run_CVA6.py without --keep-build does. The window is a compile-time define, and it has to be exported: the driver's build runs make verilate again, which drops a window given only on an earlier make verilate command line, while an exported one reaches that build too. apply keeps each file it changes beside it as <name>.upstream, which is how revert puts it back, since this container has no git history to restore from, and status says what is in.

⁠Benchmarks and configurations

Each benchmark folder carries a one-line .overhead_suite file naming the overhead table its programs belong to, so the right fixed cost is subtracted wherever the folder is copied. benchmarks/config/ is the calibration set and benchmarks/viewer/ the set written while developing the viewer.

CVA6_configs/ holds three packages, and the live copy the build elaborates is core/include/cv64a6_imafdc_sv39_hpdcache_wb_config_pkg.sv:

A sweep writes its package over the live one, runs, and restores the live one afterwards, on success, on failure and on interrupt. A sweep killed outright leaves core/include/cv64a6_imafdc_sv39_hpdcache_wb_config_pkg.sv.sweep_backup behind, and the next sweep restores the live package from it first. Until then a plain run_CVA6.py builds whichever configuration the killed sweep had installed, and warns about it, so copy the backup back by hand if a run comes first.

⁠Cleaning up

python3 scripts/clean_CVA6_runs.py --dry-run    # list, delete nothing
python3 scripts/clean_CVA6_runs.py              # list, ask about work-ver, then ask

It deletes results/run/, results/batch/, results/sweep_CVA6Flow/, results/sweep_CVA6_config/, results/verif/, the dated verif/sim/out_* folders, work-ver/ and every __pycache__ it finds. work-ver is asked about separately because a run that leaves the configuration alone can reuse it, and --keep-build spares it without asking.

⁠What is upstream and what is not

The core, its SoC wrapper, the verification harness and the vendored dependencies are the work of the OpenHW Group and contributors, under the licences preserved in the image: LICENSE, LICENSE.Berkeley and LICENSE.SiFive.

What this project adds is listed in LICENSE.FaMAF, which also carries the MIT terms it is offered under. CITATION.cff is how to cite the image. The diffs in verilator_changes/ change upstream files, and once scripts/patch_CVA6_testbench.py puts one in, each file it edited carries a notice of what was changed, as their licences require.

⁠Built with

riscv-none-elf-gcc 13.1.0, Verilator 5.008, Spike 1.1.1-dev, Python 3.12 on Ubuntu 24.04, the same base as the gem5 image. The toolchain is built from the fork's own util/toolchain-builder, and Verilator and Spike from the harness's install scripts, so the versions are the ones the flow expects rather than whatever a distribution ships.