manuel313/famaf_cva6 repository overview

⁠The CVA6 container

The real-hardware side of the FaMAF CVA6 Project: the frozen CORE-V CVA6 fork, a built RISC-V toolchain, Verilator, Spike, the run drivers and the CVA6Flow viewer, in one image. Everything lives under /CVA6, which is the working directory of every shell the image starts.

⁠Quick start

docker pull manuel313/famaf_cva6
docker run -dit --name CVA6 -p 8001:8000 --shm-size 2g manuel313/famaf_cva6 bash
docker exec -it CVA6 bash

Then, inside the container:

python3 scripts/run_CVA6.py benchmarks/config/store_fwd.S --no-vcd   # the first run builds the Verilator model, about 2 minutes
python3 scripts/serve_CVA6Flow.py    # then open http://localhost:8001/CVA6Flow/CVA6Flow.html

The viewer's Load sample button works straight away: the image carries a full sample of each of its programs that fits the 500,000 records the page renders at once, five of the nine, made while it was built. With the repository cloned, python3 scripts/make_containers.py cva6 --pull does the same with limits sized to your machine, and scripts/docker_sync.py moves files between the checkout and the container.

⁠Tags

⁠Its counterpart

manuel313/famaf_gem5⁠ runs the same benchmarks on gem5's MinorCPU, matched to this core. The project, its documentation and the recipe for this image are at https://github.com/FaMAF-CVA6-Project/CVA6⁠, and the comparison viewer that puts the two side by side, FlowCompare, is there too.

Everything below describes what is inside the container, so every path is a container path.

⁠What is in here

The target built here is cv64a6_imafdc_sv39_hpdcache_wb, a 64-bit CVA6 with the HPDcache writeback data cache, a 16 KiB 4-way instruction cache and a 32 KiB 8-way data cache.

⁠Run one benchmark

python3 scripts/run_CVA6.py benchmarks/config/store_fwd.S --no-vcd

The first run compiles the Verilated model into work-ver/, which took about a minute and a half on a 16-thread machine. Every run after that can reuse it with --keep-build, as long as the configuration package and the testbench are unchanged.

What it prints is a metrics table with two figures per metric: OFFICIAL is what the hardware counters read, and NET subtracts the fixed instrumentation overhead of the suite the program belongs to, which is the number the gem5 side is compared against.

Output lands under results/: the files worth keeping in results/run/, and the simulator's own output folder moved to results/verif/out_<date>/ when the run survives to the end.

⁠The other drivers

Every one of them answers --help. patch_vcd_window.py and measure_CVA6_overhead.py take -n for a dry run, clean_CVA6_runs.py and run_all_CVA6_benchmarks.py take --dry-run, and both sweeps take --dry-run to print the plan without running it.

⁠Use the viewer

python3 scripts/serve_CVA6Flow.py

It listens on port 8000 inside the container, which make_containers.py publishes as 8001 on the host, so open http://localhost:8001/CVA6Flow/CVA6Flow.html there. The gem5 container takes 8000, so both can serve at the same time.

The viewer loads the JSON the tracer makes from a VCD:

python3 CVA6Flow/CVA6Flow_tracer.py results/run/daxpy.vcd -o results/run/daxpy.json

The page opens a JSON from the host's disk, dropped onto it or chosen with Load JSON, so either copy the JSON out to the host, with docker cp or the fork's scripts/docker_sync.py pull, or make a sample of it inside the container, which the served page then offers by name:

python3 scripts/make_CVA6Flow_sample.py results/run/daxpy.json -n 0

-n 0 keeps every record, and the script refuses a JSON of more than the 500,000 records the page renders at once. Without -n it keeps the first 3,000. The sample lands in CVA6Flow/tests/ as daxpy.sample.js, listed in CVA6Flow/tests/samples.js, and the page scripts/serve_CVA6Flow.py serves offers it in its sample picker.

The tracer finds the disassembly listing on its own when it sits beside the VCD under the same name, and --disasm-list names it otherwise. With --strict the tracer exits with 3 on a degraded VCD: one missing the signals of a mechanism the tracer follows, so that mechanism could not be observed at all, or one that was cut, holds no value changes, no rising clock edge, no instruction or no commit, starts part way through the run, as a windowed dump does, or ends at a timestamp that disagrees with its cycle count. metadata.degraded always names each one, whether or not --strict is passed. The viewer refuses a JSON whose metadata.schema_version is not 3 and asks for it to be regenerated, rather than drawing silent nulls.

CVA6Flow/tests/ already holds a full sample of every program in benchmarks/viewer/ whose run fits the 500,000 records the page renders at once, every record of it, made while this image was built, so Load sample works before anything has been run here. Four programs run past that limit and have no sample: commit_ilp_test, data_cache_stress_test, icache_footprint_test and store_miss_test. The page offers only what tests/samples.js lists that its tracer wrote at schema 3, so a sample made later appears too once its .sample.js and the manifest are in CVA6Flow/tests/. On the host, python3 scripts/docker_sync.py push CVA6 replaces the folder with the checkout's own.

⁠Windowed waveforms

An unwindowed VCD of a long benchmark reaches tens of gigabytes. The two modified copies in verilator_changes/custom_size_vcds/ bound the dump to a window of clock cycles, whose VCD timestamps are twice the cycle numbers:

python3 scripts/patch_vcd_window.py apply
export trace_start=100000 trace_end=200000
python3 scripts/run_CVA6.py benchmarks/viewer/daxpy.S
python3 scripts/patch_vcd_window.py revert

The window is a compile-time define, and it has to be exported: the driver's build runs make verilate again, which drops a window given only on an earlier make verilate command line, while an exported one reaches that build too. apply keeps the file it replaces beside it as <name>.upstream, which is how revert puts it back, since this container has no git history to restore from.

⁠Benchmarks and configurations

Each benchmark folder carries a one-line .overhead_suite file naming the overhead table its programs belong to, so the right fixed cost is subtracted wherever the folder is copied. benchmarks/config/ is the calibration set and benchmarks/viewer/ the set written while developing the viewer.

CVA6_configs/ holds three packages, and the live copy the build elaborates is core/include/cv64a6_imafdc_sv39_hpdcache_wb_config_pkg.sv:

A sweep writes its package over the live one, runs, and restores the live one afterwards, on success, on failure and on interrupt. A sweep killed outright leaves core/include/cv64a6_imafdc_sv39_hpdcache_wb_config_pkg.sv.sweep_backup behind, and the next sweep restores the live package from it first. Until then a plain run_CVA6.py builds whichever configuration the killed sweep had installed, so copy the backup back by hand if a run comes first.

⁠Cleaning up

python3 scripts/clean_CVA6_runs.py --dry-run    # list, delete nothing
python3 scripts/clean_CVA6_runs.py              # list, ask about work-ver, then ask

It deletes results/run/, results/batch/, results/sweep_CVA6Flow/, results/sweep_CVA6_config/, results/verif/, the dated verif/sim/out_* folders, work-ver/ and every __pycache__ it finds. work-ver is asked about separately because a run that leaves the configuration alone can reuse it, and --keep-build spares it without asking.

⁠What is upstream and what is not

The core, its SoC wrapper, the verification harness and the vendored dependencies are the work of the OpenHW Group and contributors, under the licences preserved in the image: LICENSE, LICENSE.Berkeley and LICENSE.SiFive.

What this project adds is listed in LICENSE.FaMAF, which also carries the MIT terms it is offered under. CITATION.cff is how to cite the image. The two files in verilator_changes/custom_size_vcds/ are modified copies of upstream files and each carries a notice of what was changed, as their licences require.

⁠Built with

riscv-none-elf-gcc 13.1.0, Verilator 5.008, Spike 1.1.1-dev, Python 3.12 on Ubuntu 24.04, the same base as the gem5 image. The toolchain is built from the fork's own util/toolchain-builder, and Verilator and Spike from the harness's install scripts, so the versions are the ones the flow expects rather than whatever a distribution ships.