psvensson/lagrange repository overview

⁠Lagrange

Lagrange is a distributed runtime for data-intensive services. A cluster of equal nodes stores partitioned, Raft-replicated SQL tables; services deploy as WASM, and Lagrange runs each part of a request on the nodes holding the relevant data, so only reduced results cross the network.

• Source: https://github.com/psvensson/lagrange⁠
• Issues: https://github.com/psvensson/lagrange/issues⁠
• License: AGPL-3.0-only

⁠Tags

Images are linux/amd64 only and built on distroless Node.js 22⁠ - there is no shell, package manager, or npm inside the container.

⁠Release notes

⁠0.4.0 — 2026-10-31

Scenario A (tidb-oltp-baseline) deep-stack throughput and latency optimization, distributed scale-out, provenance separation, longitudinal metrics evolution, and unified 5-node and Scenario C summary benchmark release completing epics scenario-a-deep-stack-throughput-and-latency and distributed-scaleout-and-provenance-benchmark-series (Slices 1–5):

Formation health at cut time: formation health: 0/20 passed (0%), 0 with a starved seed.

⁠Added

• Scenario C in-partition $O(1)$ micro-bucket feature rollups & epoch-watermarked lazy backfill (partition-feature-rollup-constants.js, partition-feature-rollup-store.js, epoch-lazy-feature-backfill.js, scenario-c-rollup-comparator.js): Maintains synchronous per-entity decomposable micro-buckets (1 s / 60 s) in $O(1)$ inside the partition leader apply path (584.2 dec/s on 3-node and 2420.0 dec/s on 5-node RF=3 across 0 ms, 100 ms, and 1000 ms SLAs with 0.0% stale reads), evaluates multi-window sliding rollups (15m, 30m) in $O(1)$ with 0.0 s backfill on decomposable window/weight changes, and performs epoch-watermarked lazy on-read cutover on hot entities plus bounded in-process background replay (0 B WAN traffic) for new raw attributes.
• Locking-read bound parameter propagation & split-PK tuple resolution (sql-query-engine-select-execution.js, sql-locking-read-execution.js): Propagates bound params and queryOptions into executeLockingReadReservations, merges equality-bound WHERE PK columns with projected SELECT columns so both full-PK-in-WHERE (SELECT quantity FROM stock ... FOR UPDATE) and split-PK (SELECT order_id FROM new_order WHERE warehouse_id = ? AND district_id = ? FOR UPDATE) locking reads resolve canonical PK tuples in memory with 0 extra SELECT RPCs, and preserves orderBy, limit, and offset on any fallback PK query.
• Coordinator participant-set deduplication & transient status skip (distributed-transaction-participant-set.js, sql-query-engine-transaction-persistence-methods.js, distributed-transaction-telemetry.js): Skips redundant persistAdmittedParticipantSet gateway mutations when the admitted participant_set_text is already persisted on tx.participantSetText (tracked via skippedParticipantSetMutations), and skips non-authoritative pre-RPC participant status writes (PREPARING, COMMITTING, ROLLING_BACK) while preserving ACTIVE, PREPARED evidence, COMMITTED, ROLLED_BACK, and FAILED.
• Raft runtime proposal batching & durable batch telemetry (raft-rs-runtime-owner.js, raft-rs-durable-store.js): Coalesces concurrent propose commands arriving while a Raft group turn is in flight into a single drainReady cycle and tracks batchedEntryCount and multiEntryReadyCount in getDurableSyncStats().
• Canonical row-key extraction, disjoint-commit generation advancement & $O(1)$ disjoint write staging (partition-participant-transaction-session.js, partition-participant-transaction-request.js, partition-participant-transaction-apply.js, partition-participant-transaction-staging.js): Extracts canonical PK row keys from INSERT column lists and UPDATE/DELETE WHERE clauses, advances session.generationBase at sealSession when intervening committed generations touched only disjoint row keys, cleans up prepared participant row keys on rollback/refusal/expiry, and validates disjoint-row staged writes directly in $O(1)$ (disjointDirectValidations).
• Distributed scale-out & provenance benchmark series (#169–#173): Scenario B measurement provenance audit and reclassification (local-oracle-plus-network-model / contract-model-non-comparable) plus longitudinal Lagrange metrics evolution (lagrange-metrics-evolution.{md,html,svg}, Slice 1); 3-to-5-node RF=3 partial replica-coverage (100% vs 60%) scale-out and Scenario A three-mode separation (scaleout-3to5-node-rf3-progress.{md,html,svg}, Slice 2); 5-node RF=3 partition granularity sweep across P=6, 15, 30, 60 (partition-granularity-sweep-progress.{md,html,svg}, Slice 3); Scenario B network RTT crossover (0.2 ms to 30 ms) and working-set growth (scenario-b-rtt-working-set-crossover.{md,html,svg}, Slice 4); and live partition movement under load plus Scenario C 5x joint scaling and operating-boundary map (movement-and-scenario-c-scaling-progress.{md,html,svg}, Slice 5).
• Unified 5-node (RF=3) and Scenario C summary page statistics (0.3.3 & 0.4.0): Added 5-node RF=3 (lagrange-5node-rf3) and Scenario C (streaming-feature-risk-decision) statistics, charts, and multi-system tables directly to the benchmark summary landing page (docs/benchmarks/README.md, docs/benchmarks/index.html, and docs/benchmarks/comparative-overview.svg).

Artifacts and checksums: release page⁠.

⁠0.3.3 — 2026-10-30

Scenario A transaction coordination fast-path and multi-system comparative benchmark completion release reducing per-statement coordinator metadata bookkeeping, overlapping independent participant BEGIN/PREPARE/COMMIT/ROLLBACK stages, adding opt-in row-key-disjoint concurrent participant staging sessions, reusing pre-validated compiled operations across staging replay, pruning composite-key partitions by prefix and range, caching SQLite determinism compilation plans, and completing Slices 3–5 of the comparative benchmark matrix (epic scenario-a-transaction-coordination-fast-path):

Formation health at cut time: formation health: 0/20 passed (0%), 0 with a starved seed.

⁠Added

• Scenario A transaction coordination fast path & telemetry (scenario-a-transaction-coordination-fast-path): Per-transaction phase and metadata telemetry (distributed-transaction-telemetry.js), reduced durable transaction bookkeeping (sql-query-engine-transaction-set-methods.js, distributed-transaction-participant-set.js, distributed-transaction-statement.js), concurrent independent participant BEGIN, PREPARE, COMMIT, and ROLLBACK stages (distributed-transaction-protocol.js), opt-in row-key-disjoint concurrent participant staging sessions (partition-participant-transaction-session.js), pre-validated compiled operation reuse and empty-read direct execution during staging replay (partition-participant-transaction-staging.js), composite partition prefix and range pruning (partition-resolver-composite.js, lagrange-oltp-adapter.js), and schema_version-fenced SQLite determinism plan caching plus Raft durable sync telemetry (partition-transaction-determinism.js, raft-rs-durable-store.js).
• Multi-system comparative benchmark matrix completion (#166–#168): Citus colocated tables and distributed functions comparison (Slice 3), YugabyteDB 2024.2 and Apache Ignite 3 colocated compute comparison (Slice 4), and Scenario C PostgreSQL + Valkey freshness-bounded risk comparison (Slice 5).

Artifacts and checksums: release page⁠.

⁠0.3.2 — 2026-10-30

Queryable Core 0.3.2 completion release sealing synchronous transactional global index maintenance (Mode A), end-to-end SQL key-routed read/write journeys (J2), local and compound secondary index journeys (J3), locking read conflict and timeout journeys (J4), online global secondary index lifecycle and routing journeys (J5), push-driven live-query subscription journeys (J6), PG-wire idle rollback notice preservation, comparative OLTP and Scenario B benchmark matrix progression artifacts, and solver/publisher proof reuse (epic release-0-3-queryable-core):

Formation health at cut time: formation health: 0/20 passed (0%), 0 with a starved seed.

⁠Added

• Queryable Core 0.3.2 journey completion (J2–J6, C1): Composite partition key resolution (partition-resolver-composite.js), global index lifecycle coordinator (global-index-lifecycle-coordinator.js and index-service-global-methods.js), live-query group helpers (live-query-group-helpers.js), Mode A synchronous global index maintenance specification seal (global-index-maintenance-mode-seal-guard.js), and end-to-end journey suites (J2–J6).
• Comparative OLTP and Scenario B benchmark progression & matrix (#158–#164): Version-gated TiDB and PostgreSQL comparison history store, Scenario B three-island distributed risk decision comparators and fairness gate, and static Markdown, HTML, and SVG comparison progression renderers.
• Solver & publisher proof deduplication (scripts/solve/commands.js, scripts/checks/push-gate-change-proof.js, scripts/publish-head.js): solve land records a commit- and base-bound test proof receipt (lagrange-land-proof.json) reused by the push gate, pushGatedHead skips redundant .githooks/pre-push re-entry via LAGRANGE_GATE_ALREADY_PROVED_SHA, corpus ratchets execute concurrently, and dependency-cruiser import-graph analysis caches verified reports in <gitCommonDir>/lagrange-import-graph-cache.json.

⁠Fixed

• PG-wire idle transaction rollback notice preservation (postgres-wire-adapter.js): Preserves the idle-timeout NoticeResponse (25P02 / 57P01) before closing or resetting the session state, and ensures Scenario A public PG contexts issue explicit ROLLBACK before pool release (oltp-scenario-a-lagrange-public-pg-context.js).
• Runtime container /proto asset inclusion (Dockerfile): Copies /proto into the production runtime image stage (J1 repair).

Artifacts and checksums: release page⁠.

⁠0.3.0 — 2026-10-29

Phase 0.3 Queryable Core release delivering explicit typed partition boundary ordering across split, merge, and routing paths, SQL-first local and compound index DDL and planner selection, PostgreSQL locking reads (SELECT ... FOR UPDATE and SELECT ... FOR SHARE with NOWAIT), partition-independent Global Secondary Indexes (CREATE INDEX ... GLOBAL) with online backfill and point-read routing, and generic push-driven live queries over table CDC without polling (epic release-0-3-queryable-core):

Formation health at cut time: formation health: 0/20 passed (0%), 0 with a starved seed.

⁠Added

• Typed partition boundary ordering and representation (partition-key-ordering-owner-completion, A1–A4): Canonical partition-boundary-representation.js and split-key-comparator.js enforce explicit partition_key_type (INTEGER, REAL, TEXT, BLOB) ordering across CREATE TABLE, PartitionResolver, ManagedSplitWorkflow, and ManagedMergeWorkflow with zero text-to-number coercion or untyped fallback.
• SQL-first local and compound secondary index DDL and planner selection (A5–A6): SQLQueryEngine routes CREATE [UNIQUE] INDEX [IF NOT EXISTS] and DROP INDEX [IF EXISTS] through IndexService and replicates local index DDL across table partitions; DistributedQueryPlanner and QueryOptimizer enforce strict leftmost-prefix matching for single-column and compound indexes.
• PostgreSQL locking reads (SELECT ... FOR UPDATE / FOR SHARE, B1–B5): Canonical AST normalization (lockingRead: {strength, waitPolicy, ofTables}), explicit transaction-block admission (25P01 outside BEGIN), Fail-Fast / NOWAIT conflict rejection (SQLSTATE 55P03), and participant row-lock reservations that block conflicting concurrent UPDATE/DELETE/INSERT and release deterministically on COMMIT, ROLLBACK, and session teardown.
• Global Secondary Indexes (CREATE INDEX ... GLOBAL, C1–C5): Catalog metadata contract (scope: 'GLOBAL', lifecycle state machine WRITE_ONLY -> BACKFILLING -> PUBLIC -> DROPPING), independent global index dataset and Raft partition provisioning, idempotent online backfill coordinator, synchronous write-path index maintenance across INSERT, UPDATE, and DELETE, and two-hop point-read routing when PUBLIC.
• Generic push-driven live-query data plane (D0–D6): LiveQueryService, LiveQueryManager, and LiveQueryGroup subscribe active SQL queries directly to CDCManager table events and re-evaluate and push delta updates (added, modified, removed) to WebSocket subscribers without polling timers.
• Solver parallelization and home lab placement (scripts/solve/*, scripts/lab/probe.js): solve.js land synthesizes a candidate commit from the staged index for remote lab test placement, routes select-change-tests.js through runPlacedClassifiedTestFiles(), deduplicates .githooks/pre-commit, and serializes concurrent worktree landings via a cross-worktree lock.

Artifacts and checksums: release page⁠.

⁠0.2.8 — 2026-10-29

Patch release repairing the runtime container build context for wit/ after the 0.2.7 partial-channel release (where npm published 0.2.7 before the Docker image step failed on .dockerignore) and improving release-proof receipt recording and reuse (epic release-0-2-7-stabilization):

Formation health at cut time: formation health: 0/20 passed (0%), 0 with a starved seed.

⁠Fixed

• Docker build context for wit/ (release-0-2-7-docker-context-and-publish-speedup): .dockerignore now admits !wit/ and !wit/** alongside Dockerfile's COPY wit/ ./wit/, and test/release/project-hardening-contracts.test.js enforces that the build context admits wit/ for the runtime image stage.
• Whole-corpus proof receipt recording (scripts/proof-authority.js): recordProof skips release-content identity traversal when recording non-identity-reusable contracts (corpus-full-v1), preventing cold working-tree scans from exceeding the publisher's receipt-recording timeout.
• Release-proof whole-corpus receipt reuse (scripts/run-release-proof.js): npm run check:release reuses an exact-SHA corpus-full-v1 receipt for HEAD and runs only the non-corpus CI and release-hardening steps instead of repeating npm run test:all.

Artifacts and checksums: release page⁠.

• 0.2.7 (2026-10-29) — release notes⁠
• 0.2.6 (2026-10-27) — release notes⁠
• 0.2.5 (2026-09-13) — release notes⁠
• 0.2.4-rc.2 (2026-09-12) — release notes⁠
• 0.2.4-rc.1 (2026-09-12) — release notes⁠
• 0.2.4-rc.0 (2026-09-12) — release notes⁠
• 0.2.3 (2026-09-11) — release notes⁠
• 0.2.2 (2026-09-11) — release notes⁠
• 0.2.1 (2026-09-10) — release notes⁠
• 0.2.0 (2026-09-10) — release notes⁠
• 0.1.1 (2026-08-22) — release notes⁠
• 0.1.0 (2026-07-02) — release notes⁠

⁠Safe local quick start

Bind exposed ports to host loopback. The admin listener is unauthenticated. This command is local-host only:

docker run --rm \
  -e ADMIN_WS_HOST=0.0.0.0 \
  -e ADMIN_ALLOW_INSECURE_EXTERNAL_BIND=true \
  -p 127.0.0.1:8080:8080 \
  -p 127.0.0.1:8081:8081 \
  -p 127.0.0.1:8082:8082 \
  psvensson/lagrange:latest

A node started without SEED_NODE_ADDRESS becomes the seed of a new cluster. It opens three ports:

To interact with the node, connect the admin CLI from a checkout of the repository (the image itself has no shell to exec into):

git clone https://github.com/psvensson/lagrange && cd lagrange
npm install
npm run cli -- localhost:8081

⁠Multi-node cluster

One container per node is the normal deployment model. Listener ports derive from each node's REST base unless individually overridden. Joiners point SEED_NODE_ADDRESS at the seed and advertise an address other nodes can reach:

docker network create lagrange-net

docker run -d --name seed --network lagrange-net \
  -e TRANSPORT_WS_HOST=0.0.0.0 \
  -e ADMIN_WS_HOST=0.0.0.0 \
  -e ADMIN_ALLOW_INSECURE_EXTERNAL_BIND=true \
  -e NODE_ADDRESS=seed:8080 \
  -e NODE_ADVERTISED_WS_ADDRESS=seed:8082 \
  psvensson/lagrange:latest

docker run -d --name node2 --network lagrange-net \
  -e TRANSPORT_WS_HOST=0.0.0.0 \
  -e ADMIN_WS_HOST=0.0.0.0 \
  -e ADMIN_ALLOW_INSECURE_EXTERNAL_BIND=true \
  -e NODE_ADDRESS=node2:8080 \
  -e NODE_ADVERTISED_WS_ADDRESS=node2:8082 \
  -e SEED_NODE_ADDRESS=http://seed:8080 \
  psvensson/lagrange:latest

Leave NODE_ID unset: join admission requires a UUID identity, which the node mints on first start and restores from its data directory on restart.

The transport listener is plain WebSocket without cryptographic peer authentication or encryption. Run a multi-node cluster only on an isolated private network and restrict port 8082 to known cluster nodes.

⁠Configuration

Set via environment variables (-e):

Deprecated names ADMIN_WEBSOCKET_HOST, ADMIN_WEBSOCKET_PORT, and NODE_WS_PORT are still accepted (with a startup warning) when the canonical name is unset.

⁠Persistence

All state - partition storage, Raft logs, and the node's durable identity - lives under DATA_DIR (/app/data). Mount a volume to survive container replacement:

docker run -d -v lagrange-data:/app/data \
  -e ADMIN_WS_HOST=0.0.0.0 \
  -e ADMIN_ALLOW_INSECURE_EXTERNAL_BIND=true \
  -p 8080:8080 -p 8081:8081 -p 8082:8082 \
  psvensson/lagrange:latest

⁠Kubernetes

The repository ships a Helm chart (1 seed + N joiner StatefulSets, name-first addressing, per-pod PVCs) that deploys this image: charts/lagrange-node⁠. The chart publishes REST and transport only; its unauthenticated admin listener stays pod-local on loopback and cannot be externally enabled through chart values. Its port derives from node.restPort unless explicitly overridden.

⁠Memory

The container starts Node with --max-old-space-size=1536. Give the container more than that (the Helm chart defaults to a 2 GiB limit), or override the heap cap by replacing the command:

docker run ... psvensson/lagrange:latest --max-old-space-size=3072 src/index.js

⁠Learn more

• README - what Lagrange is, mental model, examples⁠
• Architecture overview⁠
• Release process and guarantees⁠