mancube/forge repository overview

⁠forge — batch-inference coordinator

Fan a giant JSONL of prompts across N workers — any OpenAI-compatible engine (vLLM / SGLang / llama.cpp / Ollama / …) or hosted provider endpoint (OpenAI-compatible or Anthropic-Messages-compatible) — survive kills and spot interruptions, and prove completeness, as a single static binary. This image is that binary: FROM scratch, ~5 MB, no shell, no libc, nonroot. forge is the orchestration shell only — it never runs inference and never provisions instances; your engines stay where they are.

Source, docs, benchmarks: https://github.com/lucheeseng827/forge⁠ (Apache-2.0)

⁠Where it fits

forge is the coordinator between the jobs you submit and the engines that run them: it owns the work queue, sharding, lease/retry, checkpoint, and result aggregation — and nothing else. It never runs inference and never provisions GPUs; your engines stay where they are, and the boxes on the right are yours.

   CLIENTS                   FORGE                      WORKER FLEET
   (submit a batch)          (this image)               (run the inference)

 ┌──────────────┐
 │ OpenAI SDK   │  /v1/files · /v1/batches ─┐
 │ (serve-batch)│                           │
 └──────────────┘                           │    ┌──────────────┐    ┌──────────────────┐
 ┌──────────────┐                           │    │  forge       │    │ vLLM / SGLang    │
 │ forge CLI    │  run·resume·verify·cost ──┼──▶ │  queue·lease │──▶ │ llama.cpp/Ollama │
 │ (JSONL)      │                           │    │  retry·ckpt  │ ◀──│ (your spot GPUs) │
 └──────────────┘                           │    │  aggregate   │    └──────────────────┘
 ┌──────────────┐        ┌──────────────┐   │    └──────┬───────┘    ┌──────────────────┐
 │ your app     │  ────▶ │  forge-core  │ ◀─┘           │       ───▶ │ …or hosted APIs  │
 │ (embed lib)  │        │  fan-out     │               │            │ OpenAI·Anthropic │
 └──────────────┘        └──────────────┘               │            └──────────────────┘
                                                         │  OpenAI-compatible HTTP
                          results.jsonl + dead-letter  ◀─┤
                          checkpoint state.db (SQLite/redb)  ·  or S3/GCS/Azure results

• Upstream — whoever submits work: unmodified OpenAI-SDK code against the serve-batch REST front door (/v1/files, /v1/batches), the forge CLI on a JSONL of OpenAI Batch-format requests, or an app embedding forge-core. forge takes a batch, not a connection.
• forge — one static binary: shards the input into a durable queue, leases items to workers, retries/dead-letters, checkpoints every result by custom_id, and aggregates — so a kill or spot-eviction resumes with zero double-work.
• Downstream — any OpenAI-compatible engine (vLLM / SGLang / llama.cpp / Ollama) on your own GPUs, or a hosted OpenAI/Anthropic-compatible API. Results land as a matched JSONL + dead-letter sidecar locally, or in S3/GCS/Azure.

⁠Tags

Pin a version tag (mancube/forge:v0.3.0) in anything you deploy; latest moves.

⁠What's new

v0.3.0 — both servers are safe to bind beyond loopback, and the OpenAI front door does what SDK code expects, mid-run included.

• Breaking: forge-agent serve refuses a non-loopback --bind without FORGE_COORDINATOR_TOKEN (set it on the coordinator and every agent), unless you pass --insecure-no-auth. Keep agents and coordinator on the same minor.
• Authenticated coordinator wire: bearer token checked in constant time before the body is read; request bodies capped at 64 MiB.
• Mid-run results: GET /v1/batches/{id}/results streams every finished line with a tailing cursor, including from cancelled and failed batches.
• GET /v1/files and DELETE /v1/files/{id}, so files.list() / files.delete() work; batches.list() now paginates and ends.
• serve-batch reads its key from FORGE_API_KEY, keeping it out of the process list; a blank key is a startup error.
• Cancel matches OpenAI: cancelling until the run has stopped, then cancelled.
• Security: fixed unauthenticated requests that could abort or freeze either server (oversized declared bodies, chunked-body smuggling, connection resets, descriptor exhaustion) via a patched HTTP layer. It still has no read timeouts, so front any port an untrusted network can reach with a reverse proxy.
• Lower memory: finalizing a batch streams its output instead of holding it in RAM.
• Helm chart 0.3.0 passes the key as FORGE_API_KEY and defaults to this image.

v0.2.1

• Queue leasing no longer sorts the whole pending set per item: 7.7 → 2,830 items/s at 1M items.
• forge-agent run exits once the queue drains, so spot boxes are released instead of idling.
• The image ships a /data owned by its user, and an unwritable checkpoint directory says so; the quick starts below run as the directory's owner.

Full history: CHANGELOG.md⁠.

⁠Quick start — run a batch

# input.jsonl: one OpenAI Batch-format request per line, keyed by custom_id
docker run --rm --network host --user "$(id -u):$(id -g)" \
  -v "$PWD:/work" -w /work \
  mancube/forge:v0.3.0 \
  run --input input.jsonl \
      --workers http://gpu1:8000,http://gpu2:8000 \
      --engine vllm --concurrency 256 \
      --out results.jsonl --checkpoint state.db

Kill it mid-run (or lose the box) and run the same command again — it resumes: the checkpoint is the queue state, already-stored results are never re-run, and verify proves every input id reached a terminal result.

docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/work" -w /work mancube/forge:v0.3.0 \
  status --checkpoint state.db          # counts, success rate, failure mix
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/work" -w /work mancube/forge:v0.3.0 \
  verify --input input.jsonl --results results.jsonl

--user runs the container as the owner of the directory you mount. The image's own user (uid 65532) owns nothing on your machine or a CI runner, so without it forge cannot create its checkpoint there and stops with unable to open database file. Later images also say which directory is not writable.

⁠Quick start — the OpenAI Batch REST front door

docker run --rm -p 8080:8080 -v forge-data:/data -e FORGE_API_KEY \
  mancube/forge:v0.3.0 \
  serve-batch --listen 0.0.0.0:8080 --data-dir /data \
    --workers http://gpu1:8000 --engine vllm --concurrency 256

The key comes from the environment (-e FORGE_API_KEY passes your shell's value through), so it never appears in the container's process list. --api-key still works, but it would.

Unmodified OpenAI SDK code (files.create → batches.create → poll → files.content) then runs its batch flow against your own engines — with live per-item progress. The OpenAI contract only stamps output_file_id when a batch reaches completed; to read results mid-run, or from a cancelled or failed batch, use forge's GET /v1/batches/{id}/results, which streams every finished line and returns a cursor for tailing. /v1/health is the unauthenticated liveness route; everything else requires the bearer key.

⁠Kubernetes

The chart is published as an OCI artifact right beside this image:

helm install forge oci://registry-1.docker.io/mancube/forge-chart --version 0.3.0 \
  --set 'serveBatch.workers={http://vllm-0.engines.svc:8000}' \
  --set 'serveBatch.apiKey=<bearer key>'

The same chart ships in the repo (deploy/helm/forge⁠): serve-batch as a Deployment (PVC, probes, Secret-backed key) and one-shot forge run as a Job whose PVC-backed checkpoint turns pod evictions into resumes. It defaults to this image.

⁠Notes

• --concurrency must match each engine's own limit (vLLM --max-num-seqs, SGLang --max-running-requests, llama-server --parallel) — it is the ceiling of forge's adaptive (AIMD) limiter, not a wish.
• Measured coordinator footprint: ~10 MB RSS, <1% of one core while saturating a multi-node fleet — see BENCHMARKS.md⁠.
• The container needs outbound HTTP(S) to your engines only. Upstream API keys (if any) come from the environment; nothing is ever written to the image.
• Checkpoints, results, and uploaded batch files are plaintext on the mounted volume — apply volume encryption/permissions per your data class.