aa8y/duckdb-dataset repository overview

⁠DuckDB images — aa8y/duckdb-dataset

The DuckDB images follow the same one-dataset-per-image model, and — like SQLite — DuckDB is embedded, so the build inverts the server engines' pattern: rather than shipping init scripts that run at container start, the build assembles the database file and the final image carries it. Each aa8y/duckdb-dataset⁠ image carries exactly one dataset as /data/<dataset>.duckdb, built through the Dockerfile⁠ driven by manifest.yml⁠: a dataset is described either by a text SQL source (fed to the duckdb CLI to build the database) or by a prebuilt DuckDB database file (shipped as-is).

The available tags are the DuckDB column of the dataset support matrix⁠, which also lists each dataset's upstream source.

⁠Base image

DuckDB⁠ as aa8y/duckdb-dataset⁠, built on the official duckdb/duckdb⁠ image (multi-arch, pinned in the Dockerfile⁠). That image is distroless — a glibc-linked /duckdb binary and nothing else, no shell — which shapes the build (a glibc loader stage assembles the database file, since neither the Alpine builder nor the shell-less final image can) and the image's fixed CMD: with no shell to expand variables, every image opens the same /data/db.duckdb symlink, which points at that image's dataset file.

⁠Usage

Start a container and open the database with the bundled duckdb shell:

docker run -it --rm aa8y/duckdb-dataset:world

which opens /data/db.duckdb (a symlink to /data/world.duckdb) directly; at the D prompt, SELECT count(*) FROM city;. You can also run a one-off query — the image has no shell, so the arguments replace the CMD and must name the binary by its absolute path:

docker run --rm aa8y/duckdb-dataset:world /duckdb -readonly -csv -c "SELECT count(*) FROM city" /data/world.duckdb

To open a different dataset, swap world for any tag in the DuckDB column of the matrix⁠; the file is /data/<tag>.duckdb, the tag minus any stackexchange- prefix (e.g. stackexchange-beer → beer).

⁠Keeping edits, and what --rm means

These examples pass --rm, so the container — and any change you make to the database inside it — is discarded when it exits. That suits a throwaway query session, but edits do not survive it. The image is distroless (no shell to exec), so to keep working against a modified database, copy the file out of a created container and open your own copy:

docker create --name duckdb-world aa8y/duckdb-dataset:world
docker cp duckdb-world:/data/world.duckdb ./world.duckdb
docker rm duckdb-world
duckdb ./world.duckdb   # a persistent copy on the host

⁠DuckDB datasets

Sources are in the matrix⁠; the notes below are DuckDB-specific:

• chinook: built at image-build time from the same pinned vendor script as the SQLite tag (Chinook_Sqlite.sql, release v1.4.5), rewritten for DuckDB by the scripts/chinook⁠ transform hook — DuckDB's PostgreSQL-derived parser rejects the script's bracket-quoted identifiers, and it validates FOREIGN KEY references at CREATE time where SQLite tolerates the script's forward references, so brackets become double quotes (leaving data like 'BBC Sessions [Disc 1]' untouched) and the FK constraints are dropped (mirroring what the shared pgsql hooks do for other engines). CamelCase identifiers (Track, InvoiceLine), with row counts matching the other chinook tags exactly.
• employees: datacharmer's canonical large sample (6 tables + 2 views; 300,024 employees and 2,844,047 salary rows). Upstream ships no DuckDB port; scripts/employees⁠ runs its PostgreSQL one nearly as written — composite PRIMARY KEYs, the UNIQUE constraint, CHECK (gender IN ('M','F')) and both CREATE OR REPLACE VIEWs all survive — dropping only the multi-table DROP TABLE ... CASCADE and the FOREIGN KEY ... ON DELETE CASCADE constraints (DuckDB rejects the referential action outright and validates references per row, which a 4.2M-row load cannot afford). The eight data dumps are converted to per-table CSVs and bulk-loaded with COPY <table> FROM '<table>.csv', so the ~140 MB of CSV stays behind in the build and only the database ships. Row counts are exact and match the other engines.
• world, iso3166, frenchtowns, usda, pgexercises, dellstore: the same PostgreSQL dumps the SQLite tags use, run through the shared scripts/pgsql⁠ transform hook. Because DuckDB's parser is PostgreSQL-derived, that hook is a thin one next to SQLite's: it converts COPY ... FROM stdin blocks to batched INSERTs (DuckDB's COPY reads files, not inline data), transcodes the Latin-1 dumps, and removes only what DuckDB genuinely cannot run — SET, setval, GRANT, USING btree, serial, FOREIGN KEYs, PL/pgSQL. PostgreSQL casts, column types, CHECK constraints and ALTER TABLE ... ADD CONSTRAINT ... PRIMARY KEY all survive, so these tags keep their primary keys where the SQLite ones cannot. Row counts match the SQLite tags exactly. One cosmetic difference: frenchtowns keeps the dump's declared Regions / Departments / Towns spelling, since DuckDB is case-preserving — and case-insensitive, so select * from regions resolves anyway.
• northwind, sportsdb: the same Yugabyte PostgreSQL-dialect dumps the CockroachDB tags use, through the same shared scripts/pgsql⁠ hook — a different upstream from the SQLite northwind tag (a prebuilt, much larger expanded edition), so northwind here matches the CockroachDB tag's 14 tables and counts exactly, and sportsdb matches the SQLite tag's 107. Two DuckDB-specific wrinkles: the hook drops CREATE DOMAIN (sportsdb declares primary_id AS integer and never uses it), and northwind's 17 empty bytea literals (categories.picture, employees.photo) are rewritten from PostgreSQL's '\x', which DuckDB rejects, to '' — the same empty BLOB. That rewrite is opt-in, enabled by the thin scripts/northwind⁠ hook, because the shared hook sees no column types and the identical token in a text column is the two-character string \x. northwind keeps all 14 primary keys; sportsdb declares none upstream, and its UNIQUE constraints go the way the SQLite tag's do (DuckDB has no ALTER TABLE ... ADD UNIQUE), while all 98 indexes survive.
• moma: the MoMA research collection, published only as CSV, so the schema is authored in-repo (scripts/moma⁠) and the CSVs are bulk-loaded with COPY <table> FROM '<file>.csv' (FORMAT CSV, HEADER) — DuckDB reads CSV in plain SQL, so unlike the SQLite tag no dot-commands are involved. Same two tables (artists, artworks) and column order as the other engines. Counts drift as MoMA refreshes its exports, so the smoke test records floors.
• geonames: GeoNames' cities15000 export, published only as a zipped tab-separated file, so the schema is authored in-repo (scripts/geonames⁠) and the export is bulk-loaded with a single COPY cities FROM 'cities15000.txt'. Quote processing is switched off (QUOTE '', ESCAPE '') — the export has no quoting, so a double quote in a place name is a literal character — and NULLSTR '' makes a blank field NULL, which the mostly-empty elevation column needs. Coordinates are decimal(11, 7) (the MySQL tag's scale) rather than the PostgreSQL tag's bare numeric: DuckDB's bare decimal is decimal(18, 3), which would round them to three places. Counts drift as GeoNames rebuilds the export daily, so the smoke test records floors.
• openflights: the three OpenFlights data files, schema authored in-repo (scripts/openflights⁠) and each bulk-loaded with one COPY <table> FROM '<file>.dat' (FORMAT CSV, HEADER false, NULLSTR '\N') — they are already RFC4180 CSV, so unlike the SQLite tag no post-load rewriting of the \N sentinel is needed. Same three tables and column order as the other engines, and no foreign keys (routes deliberately dangles). Counts drift as upstream edits the files in place, so the smoke test records floors.
• airlines: the postgrespro flight-bookings demo — 10,702,083 rows across nine tables, by a wide margin the largest dataset here, from the same pinned demo-20250901-3m dump the PostgreSQL and SQLite tags use. The scripts/airlines⁠ hook enables four opt-in knobs on the shared scripts/pgsql⁠ transform. The one that matters most is PGSQL_COPY_CSV: each COPY ... FROM stdin block is written out as a sibling <table>.csv and replaced by DuckDB's file-based COPY ... FROM '<file>' (FORMAT CSV, ...), so the rows arrive through the vectorized CSV reader instead of ~700 MB of INSERT text through the SQL parser — the load step takes seconds. The others flatten the bookings schema, drop the three convenience views (their bodies need AT TIME ZONE on a per-row timezone name, the range containment operator and a PL/pgSQL function) together with the COMMENT ON statements attached to them, and map the four types DuckDB has no equivalent for: jsonb → json (so SELECT model ->> 'en' FROM airplanes_data still works), and point / tstzrange / integer[] → text, each keeping the dump's literal verbatim. All nine tables keep their primary keys and column comments. Row counts match the PostgreSQL and SQLite tags exactly.
• nyc-taxi: the NYC TLC yellow-taxi trip records — 4,322,960 rows in one trips table, and the only dataset here that ships as Parquet, the format DuckDB exists to read. The scripts/nyc-taxi⁠ hook authors a single CREATE TABLE trips AS FROM read_parquet(...) (Parquet support is compiled into the CLI, so no extension is loaded), and the Parquet file itself stays behind in the build — only the database ships. The month is pinned (yellow_tripdata_2025-06.parquet) because TLC publishes monthly and revises older months in place, so an unpinned URL would make the row count drift. Column names and types are the TLC's own.
• sakila: the MySQL DVD-rental sample (the original of pagila), from jOOQ's multi-dialect port collection⁠ — its PostgreSQL flavour, by far the richest PostgreSQL dump here. It does not symlink straight to scripts/pgsql: the scripts/sakila⁠ hook first strips what that shared hook has never had to see — a CREATE DOMAIN, a user-defined aggregate, eight functions (three with a terminator the shared hook's regex misses), fourteen triggers, six INHERITS partition children with their rules and indexes, the CACHE clause on every sequence, and a USING gist index — and maps three columns DuckDB has no type for: release_year back to integer with the dropped domain's 1901..2155 CHECK re-expressed as a column constraint, fulltext from tsvector to text, and special_features from text[] to text. All seven views survive (group_concat is DuckDB's built-in, so its separator is , rather than , ). Dropping the payment_p2007_NN children costs no rows — the COPY data goes to payment itself — so the tag ships the same 15 base tables and counts as the CockroachDB sakila tag.
• stackexchange-<site> (beer, chess, coffee, cooking, poker, woodworking, outdoors, boardgames): converted from each site's per-table XML dump by scripts/stackexchange⁠, as on the other engines. The DuckDB emitter differs from the SQLite one in two places: no SQLite PRAGMA preamble (DuckDB rejects those settings), and CreationDate and friends are real TIMESTAMP columns rather than ISO 8601 text — matching the PostgreSQL/MySQL/CockroachDB tags, so date comparisons compare dates. Counts drift as archive.org refreshes the dumps, so the smoke test records floors.

Not ported: adventureworks and omdb, whose upstreams are PostgreSQL-specific (a Python-reformatted multi-schema port with materialized views, and views that depend on the tsm_system_rows extension) — see the notes in postgres/README.md⁠.

⁠Custom images

Each image carries one dataset, selected with the DATASET build arg along with that dataset's sources (declared per tag in manifest.yml⁠). The simplest way to build a tag is through dave:

dave build -c duckdb -t chinook

To add or change a DuckDB dataset, declare its extractUrl, sqlFiles (or dbFile for a prebuilt database) and any extras under a new tag in manifest.yml — the ETL Dockerfile⁠ reads them as build args. See docs/building.md⁠ for the full build instructions and how the build cache works.