Sign inSign up

cenk1cenk2/hermes-bridge-runs

By cenk1cenk2

•Updated 8 days ago

Brokers the Hermes runs API and holds jobs across Hermes restarts.

Image
0

571

cenk1cenk2/hermes-bridge-runs repository overview

⁠hermes-bridge-runs

A broker in front of a Hermes⁠ API server that speaks its own /v1/runs API. Every create becomes a job with an id of its own, held in Redis while Hermes is busy or down and resubmitted when a run is interrupted, so a caller that polls the job keeps getting an answer through Hermes restarts and broker rollouts. Talks directly to any deployment that exposes /v1/runs, and authenticates nobody: the gateway in front of it does.

⁠Image

docker run --rm -p 8080:8080 cenk1cenk2/hermes-bridge-runs:latest
TagSource
latestThe default branch.
v<version>The release @kilic.dev/hermes-bridge-runs@<version> created by semantic-release.

⁠Environment

Durations are milliseconds unless noted.

VariableDefaultDescription
PORT8080Port the HTTP server binds to.
LOG_LEVELinfoLowest level logged: error, warn, info, debug or verbose.
LOG_FORMATjsonjson for one JSON object per line, text for readable local lines.
REDIS_URLrequiredRedis or Valkey holding every queue and all shared state.
QUEUE_PREFIXhermes-bridge-runsPrefix of every Redis key and queue.
QUEUE_RETENTION_COMPLETED3600Seconds a completed job is kept.
QUEUE_RETENTION_FAILED86400Seconds a failed job is kept.
QUEUE_TRANSACTION_RETRIES10Attempts of a Redis transaction that keeps conflicting.
RUNS_RETENTION86400000Lifetime of a job, its idempotency key and its event stream after their last change.
RUNS_BODY_LIMIT4194304Largest create body in bytes.
RUNS_PROXY_ENABLEDfalseAnswer GET /v1/runs/{id} for an id no job has with the run of that id on Hermes, for runs created on Hermes before the broker took over.
RUNS_EVENT_MAX_LENGTH1000Events kept per job, trimmed from the oldest.
RUNS_EVENT_BLOCK5000Longest wait of an event stream for its next event before it checks whether its client is still there.
HERMES_URLrequiredBase URL of the Hermes API server, e.g. https://hermes.example.com/v1.
HERMES_API_KEYrequiredKey sent to the Hermes API server.
HERMES_INSTRUCTIONSnoneInstructions for a create whose body carries none.
HERMES_EVENTS_IDLE_TIMEOUT120000Silence on a run event stream before it is dropped and the run is polled.
DRIVER_SEEN_TTL86400000Lifetime of a seen idempotency key.
DRIVER_QUEUE_LOCK_TTL30000Lifetime of a thread lock.
DRIVER_QUEUE_LOCK_RENEW_INTERVAL10000Renewal interval of a held thread lock.
DRIVER_QUEUE_SWEEP_INTERVAL1000Interval of the sweep for threads with due work.
DRIVER_BACKOFF_INITIAL5000First retry delay while a run create gets 429, a 5xx or no connection.
DRIVER_BACKOFF_MAX60000Largest retry delay.
DRIVER_POLICY_BATCHfalseUnused: every job is one run of its own.
DRIVER_POLICY_STEERfalseUnused: a job is never steered.
DRIVER_POLICY_RESUBMIT_MAX0New runs for a job whose run ends interrupted, when the create has no X-Bridge-Resubmit.
DRIVER_POLICY_BACKOFF_WINDOW600000Time a job is held while Hermes is busy or unavailable before it fails, when the create has no X-Bridge-Hold.
DRIVER_POLICY_APPROVALdeny-stopdeny-stop stops a run after denying its approval request, deny-continue lets it go on.
DRIVER_FOLLOW_LEASE_TTL30000Lifetime of a run lease; another replica takes the run over once it expires.
DRIVER_FOLLOW_LEASE_RENEW_INTERVAL10000Renewal interval of a held run lease.
DRIVER_FOLLOW_SWEEP_INTERVAL5000Interval of the sweep for runs nobody follows.
DRIVER_FOLLOW_POLL_INTERVAL5000Interval between polls of a run without a stream.
DRIVER_FOLLOW_RETENTION86400000Time a finished run's record is kept.
DRIVER_HEARTBEAT_IDLE1200000Silence in a thread before it reports idle.
DRIVER_HEARTBEAT_SWEEP_INTERVAL60000Interval of the idle sweep.

⁠Routes

Errors answer the Hermes envelope {"error":{"message","type","code"}}.

MethodPathDescription
GET/healthzLiveness: answers 200 ok.
GET/readyzReadiness: answers 200 ok, and 503 once a shutdown starts draining.
POST/v1/runsCreates a job from {"input","instructions"?,"session_id"?} and answers 202 {"run_id","status","replayed"}. With an Idempotency-Key, the same key and body answer the same run_id with replayed: true and Idempotency-Replayed: true on any replica, and another body answers 409 idempotency_key_conflict. 400 on a body that fails the schema, a malformed Idempotency-Key (invalid_idempotency_key) or a malformed X-Bridge-* header (invalid_header).
GET/v1/runs/{id}The job as a Hermes run: {"object":"hermes.run","run_id","status","session_id","created_at","updated_at","output","error","usage","attempts","held_since"}, times in seconds. 404 run_not_found for an unknown id, unless RUNS_PROXY_ENABLED hands it to Hermes.
GET/v1/runs/{id}/eventsServer-sent events of the job in the Hermes vocabulary, each with an id:; a Last-Event-ID header resumes after that event on any replica. Ends after run.completed, run.failed, run.cancelled or run.interrupted, and when the replica drains.
POST/v1/runs/{id}/stopStops the job and answers it with status: stopping; a finished job answers as it is.
anyany otherAnswers 404.

⁠Runs

The broker enqueues every create as a durable job in Redis instead of calling Hermes inline: a create answers immediately with the job id, the job survives Hermes restarts and rollouts, and RUNS_PROXY_ENABLED additionally exposes runs that were created on Hermes directly. RUNS_EVENT_BLOCK bounds how long an event stream waits for its next event before it verifies the client is still connected; raise it on high-latency links if streams close early.

Every job is a thread of its own, runs-<run_id>, that never batches or steers, so one job is one Hermes run at a time and a stop stops only that job. The job id is the idempotency key of its first Hermes create and <run_id>#<attempt> of each resubmit. A session_id in the body is passed to Hermes as the session of the run and nothing else.

HeaderDefaultDescription
X-Bridge-HoldDRIVER_POLICY_BACKOFF_WINDOWMilliseconds the job is held while Hermes is busy or unavailable before it fails.
X-Bridge-ResubmitDRIVER_POLICY_RESUBMIT_MAXNew runs for the job when its run ends interrupted.
Jobstatus
Created, not yet taken by Hermes, or held while Hermes is busy or unavailable (held_since set)queued
Taken by Hermes, resubmitted after an interrupt, or stopping after a denied approvalrunning
Finishedcompleted with output as a string, or failed, cancelled or interrupted with error

attempts counts the Hermes runs of the job. A job, its idempotency key and its events are kept for RUNS_RETENTION after their last change.

⁠Logs

Every line carries a message that is a complete sentence fixed per event, its source class as context, and its ids, numbers and reasons as fields at the root of the JSON object ({"level":"log","message":"Created a job.","context":"RunsJobService","job":"job_...","key":"...","hold":2400000,"resubmit":3}); text prints the same fields inline. Durations are durationMs; bodies are never logged.

  • info: one line per HTTP request (method, path, status, duration, remote address; /healthz and /readyz only at debug), each created, replayed, held, accepted, stopped and finished job, each proxied run, run creation, busy backoff, follows and takeovers, finished runs with status and duration, Redis connections, the drain on shutdown and the runs, threads and event streams it hands over or ends.
  • debug: every appended job event, opened and closed event stream, run stream event, Hermes call (method, path, status, duration), thread lock and run lease acquire, renew and release, and Redis transaction retry.
  • warn and error: anything retried, rejected, dropped or failed, with its error.

⁠Development

The application lives in apps/runs of the workspace and runs on the shared core in packages/core; see the repository README.

pnpm install
pnpm lint
pnpm test
pnpm build
pnpm --filter @kilic.dev/hermes-bridge-runs start

Tag summary

Content type

Image

Digest

sha256:d3b4d5678…

Size

74.3 MB

Last updated

8 days ago

docker pull cenk1cenk2/hermes-bridge-runs