playgali/ps5-inference-gateway repository overview

⁠PS5 Inference Gateway

A stability-aware, OpenAI-compatible inference gateway for PS5LM Plus. Run the gateway on a Raspberry Pi or an x86_64 Docker host; the PS5 runs the language model. The gateway authenticates clients, serializes generation, measures readiness, bounds context/output, and checkpoints explicit managed tasks in SQLite.

The Docker image is public: playgali/ps5-inference-gateway⁠. The GitHub source repository is currently private while it is prepared for a future public release.

flowchart LR
    C[OpenAI-compatible clients] --> G[Gateway API :8092]
    G --> S[Durable single-flight scheduler]
    S <--> D[(SQLite WAL: tasks, steps, results)]
    S --> P[PS5LM Plus: inference]
    P -. health / slots / model fingerprint .-> G
    E[ps5-exporter: CPU telemetry] -. read only .-> G

⁠Intended experience and current boundary

The intended user experience is a regular API proxy: submit a long, complex request from a coding client, let the gateway split it into manageable PS5 calls, and receive an assembled response without manually managing backend context. The desired outcome is a Codex-like workflow supported by orchestration, not larger or riskier individual PS5 calls.

Current implementation: ordinary /v1/chat/completions requests automatically enter durable orchestration when the rendered prompt exceeds the active per-call input limit or the requested total output exceeds2,048tokens. Clients keep the normal API endpoint. Original requests remain retrievable; the gateway splits source material, runs bounded steps and returns a standard ChatCompletion or final SSE stream. Requests that already fit remain direct forwards. The explicit /v1/tasks API remains available for callers who want task-oriented control. See transparent API orchestration⁠ for limits and protocol details.

⁠What it does

• Provides the text Chat Completions subset of the OpenAI API, including SSE streaming, tools/tool results and multi-turn messages.
• Allows one backend inference at a time, even when several clients submit work.
• Checks the fully rendered chat template and exact tokenizer count before dispatch; requests are rejected rather than silently truncated.
• Splits explicit managed assignments into semantic units and a bounded dependency graph, preserving original documents, step results and rejected attempts.
• Persists queued work and idempotency keys. An interrupted, possibly dispatched inference is recorded as uncertain, not blindly replayed.
• Uses independent backend health, slot status, model fingerprint and exporter telemetry before admitting generation.
• Applies context-aware throughput monitoring, thermal admission holds and bounded recovery probes without changing PS5 settings.
• Publishes tested ARM64 and AMD64 images and supports gateway-only, drained updates following latest.

Generated code/tool calls are data. This gateway does not execute model-generated shell commands, run a Mac coding agent, install PS5 payloads, change fans, or expose a remote host-execution API. Model-assisted requirement coverage is not a compiler or proof that generated code works.

⁠Supported baseline and platforms

The present safety checks intentionally target the validated baseline:

This is not a generic arbitrary-model proxy. The runtime fingerprint checks the model, context, slot count and chat template. A mismatched backend fails closed. Supporting another model/runtime requires validating its operating envelope and updating the fingerprint/policy deliberately, rather than disabling checks.

Use a 64-bit Raspberry Pi OS for ARM64 (the production host is a Pi5). A 32-bit Pi OS image is not currently tested/published. Intel/AMD computers use AMD64; Docker Desktop/WSL2 must run Linux containers. The gateway does not contain or download model weights.

The existing production deployment is called ps5lm-gateway in Compose. That historical project/volume name is deliberately retained to preserve its database; the repository and registry image are named ps5-inference-gateway.

⁠Quick start

Prerequisites: Docker Engine with the Compose plugin, a reachable supported PS5LM Plus server, a reachable ps5-exporter endpoint, and a private LAN address on the Docker host. The PS5 runtime must already be configured; the gateway does not tune it.

git clone [email protected]:playgali/ps5-inference-gateway.git
cd ps5-inference-gateway
cp .env.example .env
mkdir -p runtime
chmod 700 runtime

Generate a bootstrap token and write only its SHA-256 digest to .env. This prints the digest, not the token:

python3 - <<'PY'
import hashlib, os, secrets
from pathlib import Path
os.umask(0o077)
token = secrets.token_urlsafe(32)
Path('runtime/bootstrap-token').write_text(token)
digest = hashlib.sha256(token.encode()).hexdigest()
p = Path('.env')
lines = [line for line in p.read_text().splitlines()
         if not line.startswith('GATEWAY_API_KEY_HASH=')]
p.write_text('\n'.join(lines + ['GATEWAY_API_KEY_HASH=' + digest]) + '\n')
p.chmod(0o600)
print('Configured SHA-256 digest:', digest)
PY

Edit .env to set the gateway host's bind address and the PS5 address. These are different machines:

# Examples only: replace both addresses for your LAN.
GATEWAY_LAN_IP=192.168.1.50
PS5_HOST=192.168.1.99
GATEWAY_IMAGE=playgali/ps5-inference-gateway
GATEWAY_IMAGE_TAG=latest
GATEWAY_PROFILE=balanced
# Keep the generated GATEWAY_API_KEY_HASH line.

Then pull and start the prebuilt image:

docker compose pull
docker compose up -d --no-build
docker compose ps
curl http://192.168.1.50:8092/healthz

The public image can be pulled without Docker Hub login. Docker selects ARM64 or AMD64 automatically. For a local source build, use docker compose build before docker compose up -d.

Keep .env, raw tokens and runtime/ private and outside Git. The bootstrap token is an administrator credential; create a scoped client credential for regular applications through POST /admin/credentials. Newly created client keys are shown once, and only their hashes are stored.

⁠Configuring the PS5 address

There is no fixed LAN IP in the runtime. Choose one of these approaches:

# Shell configuration overrides PS5_HOST in the Compose .env file.
PS5_HOST=192.168.1.99 docker compose up -d --no-build

Or set PS5_HOST=ps5.example.lan in .env. It derives:

• inference API: http://<PS5_HOST>:8081
• exporter: http://<PS5_HOST>:9100/metrics

For custom ports, paths, HTTPS or a separately hosted exporter, use full URL overrides:

PS5_UPSTREAM=http://ps5.example.lan:8081
PS5_EXPORTER=http://ps5.example.lan:9100/metrics

Nonempty PS5_UPSTREAM and PS5_EXPORTER take precedence over PS5_HOST independently. Remove old full-URL overrides if you want a new PS5_HOST to control those endpoints. IPv6 host addresses are supported; full-URL overrides use standard bracket notation. The legacy full-URL configuration remains supported for existing installations.

Important variables:

The production database lives at /data/gateway.sqlite3 in the named volume. Container address space is capped at384MiB by the launcher as a fallback for Pi kernels without memory cgroups. Compose also specifies CPU, memory, PID, tmpfs and log-rotation limits. These host resource limits do not configure PS5 memory or cooling.

⁠API examples

Set the API base and load your private token in your shell. Avoid putting a literal token in scripts or source control:

export GATEWAY_BASE=http://192.168.1.50:8092
export GATEWAY_TOKEN="$(cat runtime/bootstrap-token)"

curl "$GATEWAY_BASE/v1/models" \
  -H "Authorization: Bearer $GATEWAY_TOKEN"

curl "$GATEWAY_BASE/v1/chat/completions" \
  -H "Authorization: Bearer $GATEWAY_TOKEN" \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: first-example-001' \
  -d '{"model":"ps5lm-plus","messages":[{"role":"user","content":"Explain a bounded work queue."}],"max_tokens":512}'

For SSE, add "stream":true and use curl -N. Long orchestrated jobs send waiting heartbeats, then final content; allow suitable client timeouts. The job deadline is bounded at one hour. Use max_tokens:8192 or max_completion_tokens:8192 for a larger assembled text/code response without enlarging a PS5 call. The client model aliases include ps5lm-plus and qwen3.5-9b. A normal OpenAI-compatible client points its base URL at $GATEWAY_BASE/v1 and uses a scoped gateway client key, not an OpenAI API key.

The API is the supported Chat Completions subset, not Responses API, embeddings, images, audio, multimodal input or arbitrary llama-server runtime flags. n>1, unsupported settings, thinking-enabled requests and invalid schemas are rejected. See API reference⁠.

⁠Managed assignments

Ordinary oversized chat is automatically orchestrated. Submit a managed task when you want an immediate durable task ID and explicit asynchronous planning/status control:

curl "$GATEWAY_BASE/v1/tasks" \
  -H "Authorization: Bearer $GATEWAY_TOKEN" \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: service-design-001' \
  -d '{
    "objective":"Design a small durable job service and split the work into bounded components.",
    "requirements":["Use SQLite WAL for queued jobs.","Require bearer authentication."],
    "documents":[{"name":"notes.txt","kind":"reference","text":"The client runs on a private LAN."}],
    "max_step_output_tokens":1536
  }'

The response contains a durable task ID. Retrieve GET /v1/tasks/<id> for state, attempts, step definitions, results and errors. GET /v1/tasks/<id>/original preserves the original assignment; /sources/<source_id> retrieves preserved semantic units. Cancel with POST /v1/tasks/<id>/cancel.

Managed work is bounded: request body256KiB, at most64 semantic units,64 caller requirements/128 total extracted requirements, and1–6 topologically ordered work steps per assignment. Documents may be specification, source_code or reference; only explicit normative requirements are promoted. Work-step assignment views project the requirements assigned to that step, while originals and full dependency results remain durable. An unsplittable source unit, oversized rendered step, invalid plan or incomplete coverage fails with retained evidence rather than overflowing the backend.

States are queued, running, completed, failed, cancelled, uncertain. A completed managed assessment means its bounded model-assisted checks passed; it does not mean code compiled or tests executed. Human/worker review remains necessary. A known validation/context failure may be explicitly resumed through /resume; uncertain delivery cannot be automatically resumed or replayed.

⁠Health and operations

HTTP200 from /healthz is not proof that inference is ready. Startup/recovery requires safe independent samples. Busy slots, unavailable telemetry, fingerprint mismatches, cooldown/recovery and maintenance can prevent admission. Observe the reason rather than repeatedly retrying inference POSTs.

⁠Stability limits and evidence

The operating policy comes from the measured healthy Stage1 configuration, not older degraded runs:

• 15-minute fixed-prompt run:54.13 generated tokens/s.
• 30-minute mixed run:48 successes, no inference failures,53.70 tokens/s.
• Separate long-context cases around8K,16K,24K and28K passed; this did not establish indefinite large-context stability.
• A later real native-app coding workload used70 managed inference dispatches over~14.4minutes, with no overlaps, prompts<=2673tokens and outputs capped2048. Generated code still needed substantial review/corrections.

Profile admission limits:

The ordinary API accepts a total output budget up to16,384tokens. That is an orchestration/assembled-response budget, not one large PS5 generation. Each backend call is still capped2,048tokens; the table describes per-call limits. Plaintext assembly respects the requested total budget and returns finish_reason: length if necessary, while full step artifacts remain stored. Tool/structured final decisions remain bounded single calls.

Every dispatch must fit input+requested output+128reserve within32,768 context. Thinking is disabled. The performance profile has finite evidence and remains experimental for sustained larger contexts. A queue of32 and default per-client rate6/minute with burst8 are admission/resource controls, not throughput promises.

CPU gates82hold/80recover/86interrupt are provisional margins, not measured silicon throttle/safety limits. Essential telemetry expires after20seconds. A slowdown is evaluated against a matched context band over bounded generated-token windows. Fault recovery uses jittered backoff, independent observations and at most three failed bounded recovery probes; it never changes model/fan/server configuration. There is no evidence for a universally required fixed cooldown between healthy requests.

See baseline⁠, profile evidence⁠, acceptance⁠ and known limitations⁠. The initial fan/logging mitigations were validated together; their individual causal contributions were not isolated.

⁠Persistence, idempotency and uncertain outcomes

SQLite uses WAL and FULL synchronization. Task submissions are durable before acknowledgement; original material, step records and response chunks survive restart. Idempotency keys are scoped to the client credential: same key/same request retrieves the stored outcome, while different content returns409.

Inference POSTs are never blindly retried. If a connection disappears after dispatch, the PS5 might still be executing. The gateway preserves partial SSE output and records uncertainty instead of inventing completion or replaying generation. Stream cancellation does not prove the backend stopped instantly.

Use the supplied SQLite online-backup procedure. Do not copy a live database without its WAL or replace the named volume casually. A source/image rename does not require renaming the existing database volume. Storage quotas, bounded terminal retention and dispatch reserve limit growth; uncertain records are not silently discarded to reclaim space.

⁠Docker images, CI and latest deployments

GitHub Actions builds/tests both native architectures on GitHub-hosted runners, then publishes the exact tested image archives to Docker Hub. PRs never publish images. The Python base-image index and GitHub Actions are pinned by digest/SHA; Python dependencies are pinned in requirements.lock.

Published tags:

• latest: successful main-branch build; contains ARM64 and AMD64 manifests.
• sha-<12-character commit>: source-pinned multi-architecture image.
• sha-<commit>-arm64 / sha-<commit>-amd64: tested platform images.
• <version>: produced by a v<version> Git tag.
• deployed-6ff6941-arm64: preserved earlier production image for recovery/reference.

Publishing needs repository variable DOCKERHUB_USERNAME=playgali and secret DOCKERHUB_TOKEN (Docker Hub PAT with Read & Write). No PS5 credential, host SSH key or gateway API token belongs in CI. Without a publishing token, tests/artifacts still run and publishing is explicitly skipped. See pipeline details⁠.

On ninapi4g, a dedicated gateway-only systemd timer follows latest every two minutes. It pulls first, waits for a drain acknowledgement, makes a verified SQLite backup, replaces only the gateway container and checks health/backend readiness before reopening admission. Existing inference completes; queued managed work is preserved. Failed startup rolls back the image and quarantines that candidate instead of looping through disruptive deployments.

Following latest does not mean forcing an update through a busy/faulted backend. Stability takes priority. No CI runner is installed on the Pi; CI publishes images, and the Pi's narrowly scoped updater decides when deployment is safe. See automatic updates⁠ for ownership, one-time legacy adoption, rollback behavior and commands.

⁠Security and deployment boundary

The container runs as UID10001, drops capabilities, uses a read-only root and a dedicated data volume, disables forwarded-proxy trust, rotates logs and bounds CPU/PIDs/tmpfs/concurrency. It listens inside the container on8092; Compose binds that port to the explicit host LAN address. Keep the service on a trusted LAN, or add a separately configured authenticated/TLS front end before external exposure.

A public container image does not make a running API public. This project does not configure Nginx Proxy Manager, WireGuard, HAProxy, Immich or router forwarding. Backend URLs and live credentials are environment/runtime data, not baked into the image. Model output never becomes an operating-system command.

⁠Development and testing

python3 -m venv .venv
.venv/bin/pip install -r requirements.lock
.venv/bin/python -m unittest discover -s tests -v

CI also runs the suite inside the production image on each architecture, without network or a real PS5, then smoke-tests the service's UID, health and address-space cap. Failure tests cover interrupted streams, uncertainty/no replay, recovery, thermal policy, durable storage, scope/evidence validation, maintenance drain, configuration precedence and updater rollback/quarantine. Real-hardware evidence is documented separately; mocks do not establish hardware stability.

Useful references: API⁠, operations⁠, security⁠, managed-worker boundary⁠. Some historical acceptance scripts/docs refer to the original private lab; the runtime configuration described above is portable. Changes to context, fingerprint, recovery or thermal defaults need measured evidence and relevant tests.

The repository remains private for now. An open-source license and final public-source/history review are still pending; do not infer an open-source license merely from the public Docker image. Neither the gateway nor its image includes PS5LM Plus or model weights.