dartfx/qsv repository overview

⁠dartfx/qsv

High-performance, multi-architecture (linux/amd64 and linux/arm64) Docker distribution of QSV⁠, the blazing-fast tabular data-wrangling toolkit. This image serves as the foundational binary provider across the entire Data Artifex Docker ecosystem.

⁠Why This Image Exists

Upstream QSV distributes official precompiled binaries on GitHub Releases (qsv-*-unknown-linux-gnu.zip and musl.zip). However, users running Linux—especially on 64-bit ARM (aarch64 / ARM64) or environments requiring statically linked musl binaries (such as Alpine Linux, containerized pipelines, or minimal root filesystems)—quickly discover that the official Linux binaries have a significantly reduced feature set.

The official prebuilt binaries for aarch64-unknown-linux-musl, x86_64-unknown-linux-musl, and aarch64-unknown-linux-gnu intentionally omit major computational commands:

• sqlp & joinp (Vectorized SQL queries and joins powered by Polars)
• luau (High-speed embedded scripting engine)
• to (High-speed conversion to Parquet, Excel XLSX, ODS, and SQLite)
• readstat (Native parsing of SPSS, SAS, and Stata files)
• geocode & geoconvert (Forward and reverse geocoding)
• viz (Plotly data visualization and auto-dashboards)

⁠Why Official Linux Prebuilts Omit These Features

The upstream omission of these features is not a design preference, but a technical compromise dictated by CI/CD cross-compilation infrastructure:

1. Luau JIT Cache Invalidation on ARM64 Musl (__clear_cache):

   • The Luau JIT compiler dynamically emits machine instructions directly into memory and must invalidate the CPU instruction cache via __clear_cache.
   • On GNU/glibc systems, __clear_cache is provided by the C runtime. On Musl libc on ARM64, __clear_cache is supplied by libgcc, not Musl.
   • Without injecting custom low-level linker flags (-C link-arg=-lgcc), compiling Luau with Musl on ARM64 fails during link time with unresolved symbol errors. In GitHub Actions automated multi-target matrices, this causes build failures, leading upstream to disable luau on Linux Musl targets.

2. Polars & Arrow Memory Exhaustion in GitHub Actions:

   • The Polars vectorized query engine (polars-core, polars-sql, polars-arrow) is heavily templated. Compiling Polars with optimizations requires substantial memory (often 8–16+ GB of RAM) and extensive CPU time.
   • Standard GitHub Actions hosted Linux runners provide only 7 GB of RAM. Concurrently compiling Polars alongside all other crates under cross-compilation easily triggers the Linux OOM-killer (signal: 9, SIGKILL) or exceeds GitHub's job execution limits.

3. C/C++ Native Dependencies & Toolchain Fragility:

   • Commands such as readstat (wrapping C libraries for SAS/SPSS/Stata), to (format conversions via csvs_convert and spreadsheet-ods), and geocode (bundling spatial indexing trees) rely on native C/C++ compilation via build.rs.
   • Cross-compiling these diverse C/C++ dependencies for static Musl across both AMD64 and ARM64 in automated matrix builds requires complex, fragile cross-compilation toolchains (cross / musl-tools) that are prone to CI breakages.

4. Allocator Incompatibilities on ARM64 Musl:

   • tikv-jemallocator (the high-performance memory allocator utilized by QSV on macOS and x86_64 Linux) does not build cleanly with the native ARM64 Musl toolchain on standard CI runners, forcing upstream Musl artifacts to omit jemalloc and feature-gated bundles (qsvdp, qsvmcp).

To ensure CI builds complete reliably within runner limits, upstream opted for a minimal feature set on Linux prebuilts, recommending that users who need the full suite compile natively from source.

⁠The dartfx/qsv Solution

dartfx/qsv does the heavy lifting so you don't have to compile from source on every deployment:

• Dedicated Native/Emulated BuildKit Environment: Built in a dedicated Alpine Musl environment with full C/C++ toolchains (gcc, g++, musl-dev, libgcc) and native linker flags (-C link-arg=-lgcc).
• Tuned Compiler Profiles: Configured with optimized codegen units and persistent BuildKit cache mounts (/usr/local/cargo/registry and /build/qsv/target) to successfully compile Polars and all extensions without memory exhaustion.
• 100% Statically Linked: Completely self-contained with zero dynamic .so dependencies. The resulting binary runs natively on any Linux distribution (Alpine, Debian, Ubuntu, RHEL, CentOS, Amazon Linux) or directly inside a scratch container.
• All 78 Headless/Server Commands: Delivers the complete QSV toolkit across both AMD64 (x86_64) and ARM64 (Apple Silicon, AWS Graviton) architectures.

⁠Role in the Data Artifex Container Ecosystem

dartfx/qsv serves as the foundational binary provider across the entire Data Artifex Docker catalog. By compiling and publishing this standalone multi-arch static binary image first, all downstream Data Artifex containers import a guaranteed identical, 100% feature-complete QSV distribution via multi-stage builds:

• dartfx/base⁠: Copies /usr/local/bin/qsv directly from dartfx/qsv, combining QSV with Python 3.14, system utilities (curl, jq), and the core Data Artifex toolkits (ckan-toolkit, dataverse-toolkit, ddi-toolkit, rdf-toolkit, qsv-toolkit).
• dartfx/api-base⁠: Extends dartfx/base to provide high-performance FastAPI/Uvicorn microservices with access to QSV's vectorized Polars query engine.
• dartfx/n8n⁠: Embeds the static QSV binary alongside the @dartfx/n8n-nodes-qsv extension into the official n8n workflow automation platform.
• dartfx/n8n-runners⁠: Built on dartfx/base, allowing external task execution nodes to run QSV commands and Python scripts within isolated worker tasks.

⁠Feature Matrix: Included vs. Excluded Features

⁠1. Included Features (Full Analytical Suite)

dartfx/qsv enables all high-performance analytical, transformation, and server features:

⁠2. Excluded Features & Technical Rationale

The few upstream features excluded from this image are omitted intentionally to preserve container immutability, maintain a lean image footprint, and avoid GUI dependencies:

⁠Usage

⁠1. Standalone CLI Tool

Run any qsv command against your local data files by mounting your current directory to /data:

# Count rows in a CSV file
docker run --rm -v $(pwd):/data dartfx/qsv count input.csv

# Run Polars SQL queries on CSV or Parquet files
docker run --rm -v $(pwd):/data dartfx/qsv sqlp input.csv "SELECT * FROM _t_1 WHERE score > 80"

# Convert CSV to Parquet
docker run --rm -v $(pwd):/data dartfx/qsv to parquet input.csv -o output.parquet

# Generate an interactive data dashboard
docker run --rm -v $(pwd):/data dartfx/qsv viz smart input.csv --output report.html

⁠2. Multi-Stage Dockerfile Dependency

Copy the statically compiled binary into any existing base container (Alpine, Debian, Ubuntu, Distroless, or Scratch):

FROM dartfx/qsv:latest AS qsv-source

FROM debian:bookworm-slim
COPY --from=qsv-source /usr/local/bin/qsv /usr/local/bin/qsv

ENTRYPOINT ["/usr/local/bin/qsv"]

⁠Multi-Architecture Support

• linux/amd64 (x86_64)
• linux/arm64 (Apple Silicon M-series, AWS Graviton)

⁠Contact & Maintenance

Maintained by Data Artifex
Support: [email protected]⁠
GitHub: https://github.com/dataartifex⁠