dendra/dendra-job-workers repository overview

⁠Dendra Job Workers

Release 3 background workers for Dendra⁠. They perform asynchronous short- and long-running tasks such as image generation, vendor data extracts, and integration orchestration. These tasks should not block API requests or the UI. Worker processes connect to NATS and run job types dispatched by the dendra-job-scheduler⁠.

For system context, see System Design⁠ and the Job API⁠ guide. Platform services live in dendra-api-services⁠; browser UIs in dendra-web-apps⁠.

⁠Worker design

⁠Job scheduler and NATS

In dendra-api-services⁠, the job scheduler plans and manages jobs over Connect-RPC. One-shot, continuous, queued, and recurring modes are defined in proto/job/dendra/job/scheduler/v1alpha1/service.proto. Workers only execute registered job types when the scheduler starts a run; they do not implement scheduling themselves.

Each job type should still be written for the mode used in production: one-shots finish promptly with a clear outcome. Continuous jobs run repeatedly on an interval and must honor StopJob and per-run timeouts (run_context.go⁠). Queued jobs are one-shots the scheduler runs one at a time per queue to prevent race conditions.

Execution is over NATS: the job scheduler sends StartJob and StopJob and queries run status on worker subjects (e.g. dendra.v3.<workerType>.req.<namespace>.<jobType>.StartJob; see lib/go/jobworker/job_registry.go⁠). Workers publish Online/Offline and job status events back to the scheduler. Control-plane subject layout: packages/go/job-scheduler/NATS.md⁠ (api-services repo). Data pipeline and extract-specific subjects: packages/go/extract/NATS.md⁠.

Job scheduler --> NATS --> worker process --> job type

⁠Worker process vs job type

⁠Worker type, colocation, and dedicated binaries

Each binary has a worker type (backend, extract, load) used in its NATS subject prefix. On Online it announces the job types registered in that binary; the scheduler learns which worker type handles each job type from those events (EmitWorkerStatusEvent⁠).

Many job types can run in one worker binary when they share the same process-level dependencies and scaling profile. Add a new binary under packages/go/<name>/ when isolation helps: native libraries (libvips), memory, deploy cadence, or data-pipeline usage.

This monorepo is where future workers are added: new package, main.go, job-type registry, and Makefile targets.

⁠Job-type environment variables

Variables for a job type are usually prefixed with the job type name in ALL CAPS (same identifier as in registry.go, no separators):

• ExtractCampbellScientificLdmp → EXTRACTCAMPBELLSCIENTIFICLDMP_CHECKPOINT_RECORD_INTERVAL
• GenerateLogoAvatarVariantsToMinio → GENERATELOGOAVATARVARIANTSTOMINIO_ACCESS_KEY

Handlers typically resolve strings.ToUpper(jobType) + "_<SUFFIX>". Exceptions may exist for variables shared across related job types (e.g. IMAGEGEN_MAX_CONCURRENT_TRANSFORMS for all imagegen job types).

⁠Shared libraries

Protobuf types come from BSR (buf.build/gen/go/dendrascience/job/...); this repo does not host proto/ or release/ trees.

⁠Workers

⁠Backend worker

Binary dendra-backend-worker, source packages/go/backend⁠, worker type backend, output output/dendra-backend-worker⁠

Process notes: starts libvips; calls datapipeline.Setup for this binary.

Job-type env (examples): GENERATELOGOAVATARVARIANTSTOMINIO_*, GENERATEPHOTOVARIANTSTOMINIO_*, REVERSESYNC_WEB_API_V2_*; shared IMAGEGEN_MAX_CONCURRENT_TRANSFORMS.

⁠Extract worker

Binary dendra-extract-worker, source packages/go/extract⁠, worker type extract, output output/dendra-extract-worker⁠

Process notes: Campbell extract and prepare jobs call datapipeline.Setup in jobtypes/registry.go. Data-pipeline subjects: packages/go/extract/NATS.md⁠.

Job-type env (examples): EXTRACTCAMPBELLSCIENTIFICLDMP_*, PREPARECAMPBELLSCIENTIFICLDMP_*, EXTRACTHOBOTIMEFRAME_*; optional TOMLFILEEXTRACTHOBOTIMEFRAME_* (and similar) for TOML provider job secrets.

⁠Load worker

Binary dendra-load-worker, source packages/go/load⁠, worker type load, output output/dendra-load-worker⁠

Process notes: LiveLoadInfluxV1 calls datapipeline.Setup in jobtypes/registry.go. Mirror URLs are per deployment location via INFLUX_{LOC}_URLS.

Job-type env (examples): LIVELOADINFLUXV1_*, INFLUX_UCNRS_URLS, INFLUX_UCNRS_COERCE_NUMERIC_FLOAT

⁠Monitor worker

Binary dendra-monitor-worker, source packages/go/monitor⁠, worker type monitor, output output/dendra-monitor-worker⁠

Subscribes to platform change events (dendra.v3.changes.>), evaluates membership monitoring rules, reports alerts, and sends immediate transactional email notifications.

Env: MONITOR_RESEND_API_KEY, MONITOR_RESEND_FROM_EMAIL, MONITOR_EMAIL_RATE_LIMIT, MONITOR_EMAIL_MAX_RETRIES

Worker API key permissions (platform worker key used by monitor-worker):

• admin:organization
• report:alert
• create:notification_delivery, update:notification_delivery
• read:notification_preference, read:user_notification_config
• read:monitoring_rule, read:notification_template

Docs (under packages/go/monitor/jobtypes/monitor/⁠):

Platform design: dendra-api-services/docs/MONITORING.md⁠.

Before testing: see TESTING.md⁠ — seed membership rules (or CreateDefaultMonitoringRules), load notification templates via dendra-pytools/migration/notification_templates⁠, verify Resend sender domain (or use mock email when unset).

⁠Future workers

Add packages/go/<name>/ with main.go, register job types in jobtypes/registry.go, and extend the Makefile⁠ with build-<name> / run-<name> targets following backend, extract, and load.

⁠Repository layout

.
├── lib/go/
│   ├── jobworker/          # NATS job control, registry, runs
│   ├── datapipeline/       # Integration data JetStream publish
│   ├── utils/              # NATS, API clients
│   ├── authstrategies/     # Worker API keys
│   └── notify/             # Outbound channel senders (email/Resend)
├── packages/go/
│   ├── backend/            # dendra-backend-worker
│   ├── extract/            # dendra-extract-worker
│   ├── load/               # dendra-load-worker
│   └── monitor/            # dendra-monitor-worker
├── docker/                 # backend, extract, load, monitor, combined Dockerfiles
├── output/                 # Built binaries
├── sample.env
├── Makefile
└── go.work                 # Go 1.26 workspace

⁠Prerequisites

• Go 1.26 per go.work⁠ (toolchain go1.26)
• staticcheck (used by make check)
• NATS (control plane; often same cluster as api-services, plus DATA_PIPELINE_NATS_SERVERS when using the data pipeline)
• Running dendra-api-services⁠: make run-api and make run-job-scheduler with matching JOB_SCHEDULER_NAMESPACE and worker API key hashes (APIKEY_WORKER_API_SERVER_HASH, APIKEY_WORKER_JOB_SCHEDULER_HASH)

⁠Getting started

cp sample.env .env
# Edit .env: NATS_SERVERS, URLs, WORKER_*_KEY, namespace, job-type blocks as needed

make

In dendra-api-services (separate repository):

cp sample.env .env
# Configure NATS, connectors, auth, etc.
make
make run-api
make run-job-scheduler

Generate worker API keys on the API server (logged to stdout), then set hashes in api-services .env and keys in workers .env:

# In dendra-api-services, after build:
./output/dendra-api-server -init-action=keygen
# APIKEY_WORKER_API_SERVER_HASH=...  <->  WORKER_API_SERVER_KEY=...
# APIKEY_WORKER_JOB_SCHEDULER_HASH=...  <->  WORKER_JOB_SCHEDULER_KEY=...

Run workers (one or more):

make run-backend    # output/dendra-backend-worker
make run-extract    # output/dendra-extract-worker
make run-load       # output/dendra-load-worker
make run-monitor    # output/dendra-monitor-worker

⁠Build and check

Run make check before opening a pull request.

⁠Configuration

⁠.env

Copy from sample.env⁠. Loaded via godotenv in each worker main package.

Worker (process): required for any binary:

Job types: optional blocks in sample.env under each job-type prefix. Enable only what you run.

⁠API contracts

Job and worker messages are defined in the Buf Schema Registry⁠ module dendrascience/job. This repo imports generated Go from buf.build/gen/go/dendrascience/job/... (worker API under dendra/job/worker/v1alpha1). Scheduler protos and RPCs live in dendra-api-services (proto/job).

After schema changes in api-services, publish with buf push and bump dependencies in this repo.

⁠Development

Add a job type to an existing worker:

1. Implement a runner under packages/go/<worker>/jobtypes/.
2. Register it in that package’s jobtypes/registry.go (job type string must match scheduler / jobs.toml).
3. Document env vars using strings.ToUpper(jobType) + "_<SUFFIX>" unless shared with related types.
4. Add commented examples to sample.env⁠ under the ALL CAPS prefix.

Add a worker binary:

1. Create packages/go/<name>/ with main.go (copy pattern from backend or extract).
2. Add Makefile build-<name> / run-<name> targets.
3. Add a subsection under Workers in this README.

Data pipeline (extract): packages/go/extract/NATS.md⁠.

⁠Docker

Images expect the same .env / environment configuration as local runs. NATS and api-services are not included in the image.

⁠Further reading