dendra/dendra-api-services repository overview

⁠Dendra API Services

Release 3 platform services for Dendra⁠. This is a Go monorepo that builds the API server, job scheduler, and MCP server. APIs are defined in Protocol Buffers, served with Connect⁠ over HTTP (JSON and protobuf), and versioned as v3alpha1 (alpha).

For system diagrams and product context, see System Design⁠ on the docs site. For API usage, see Dendra Release 3 APIs⁠ and rpc-api-docs.dendra.science⁠.

⁠Executables

Build output: output/dendra-api-server, output/dendra-job-scheduler, output/dendra-mcp-server.

⁠Architecture

The API server handles Connect-RPC and auth sessions. It uses a datalayer and connector registry to reach configured data stores (commonly MongoDB, Influx, and MinIO in current deployments) and NATS JetStream for events and key-value state (packages/go/api/NATS.md⁠). The job scheduler exposes job RPC, loads specs from jobs.toml and integration registrations from the API server, and publishes work to NATS (packages/go/job-scheduler/NATS.md⁠). Workers in dendra-job-workers consume jobs and call back into the API server. An optional bridge to the Release 2 Web API supports file-import interoperability during migration.

⁠Code layout (packages/go/api)

⁠Datalayer

The datalayer is not tied to a single database or vendor. It defines per-domain Go interfaces (AuthV3Alpha1Layer, MetadataV3Alpha1Layer, TimeseriesV3Alpha1Layer, FileV3Alpha1Layer, DataintegrationV3Alpha1Layer, MonitoringV3Alpha1Layer, and others) that services use exclusively. Persistence lives in separate packages (datalayer/mongostore/..., datalayer/influxstore/...) and is wired at startup.

• Connector binding: each service module selects a store via SVC_<MODULE>_CONNECTOR (e.g. SVC_METADATAV3ALPHA1_CONNECTOR=mongo:metadata). The value is store:key, referencing a named entry in the connector registry.
• Named connectors: MONGO_CONNECTORS, INFLUX_CONNECTORS, MINIO_CONNECTORS, and similar env lists register one or more backends under short keys (e.g. metadata, cdfw, experimental). A configured connector represents a single managed store connection.
• Timeseries locations: physical location names are defined in api.toml under [timeseries_locations]. At startup the timeseries service registers one TimeseriesV3Alpha1Layer per configured location, each wired with SVC_TIMESERIESV3ALPHA1_<LOCATION>_CONNECTOR=store:key (e.g. influx:cdfw). The service exposes a single query plane over those layers.
• Shards and tables (api.toml): shardspaces assign time-bounded shards to a location (same name as in the timeseries service) and a table_storage_ruleset. table_storage_rulesets define physical database/table patterns: match.* for read/discovery (regex) and store.* for write (template tokens). Discovery results are surfaced as tables for mapping to datastreams and querying; callers work with tables, not storage layout. Shards can be scoped by org, time range, or other dimensions configured in api.toml, so capacity can grow by adding locations and shard rules rather than one monolithic store.
• Current implementations: MongoDB (auth, metadata, data integration, monitoring, and file metadata services), Influx datalayer (timeseries services), MinIO (object layer for files). These are configured at deploy-time.
• Extension: add a connector type in conns/, implement the relevant *Layer interface, and wire it in the service module’s Setup function.

Services speak datalayer interfaces; which physical store backs each domain is configured per deployment.

⁠Job scheduler (packages/go/job-scheduler)

RPC service and job controller. Job definitions come from jobs.toml (TOML file provider) and from live integration registrations via the Platform API (jobproviders/integrationsvc). System jobs in jobs.toml include change-feed subscribers for metadata reverse sync and platform side effects. NATS subject layout and worker coordination are documented in NATS.md⁠.

⁠Release 2 interoperability

services/dataintegration/v2/ bridges file-import flows to the legacy Web API v2 client (clients/webapiv2/). Release 2’s API server is not in this repository.

⁠Repository layout

.
├── api.toml              # Integration types, shards, table storage rules, pint catalog
├── defs/                 # Extra pint-go definition files loaded by api.toml [[pint.load]]
├── jobs.toml             # Preconfigured job specs for the scheduler TOML provider
├── sample.env            # Environment variable reference (copy to .env)
├── buf.yaml / buf.gen.yaml
├── docs/                 # Domain design + Go handler conventions (see docs/README.md)
├── go.work               # Go 1.26 workspace: lib/go, api, job-scheduler, mcp
├── docker/               # api, job-scheduler, mcp, combined Dockerfiles
├── lib/go/               # Shared Go library
├── packages/go/
│   ├── api/              # API server
│   ├── job-scheduler/    # Job scheduler
│   └── mcp/              # MCP server (agent tools + docs resources)
├── proto/
│   ├── api/              # Platform API → buf.build/dendrascience/api
│   └── job/              # Job API → buf.build/dendrascience/job
└── release/go/           # Generated Go (committed; from buf generate)

Protobuf schemas are the API contract; Go is the implementation language in this repo.

⁠Prerequisites

• Go 1.26 (go.work uses toolchain go1.26)
• staticcheck (used by make check)
• Buf CLI (to regenerate protos)
• Runtime services for a full local stack: see sample.env (NATS, MongoDB, MinIO, Influx as configured, IdP via Auth0 or Keycloak)

⁠Getting started

cp sample.env .env
# Edit .env: connectors, auth, NATS, API keys, etc.

make

Run both servers (use distinct ports; sample.env sets the scheduler to 8081):

make run-api              # default API_SERVER_ADDRESS :8080 if unset
make run-job-scheduler    # set JOB_SCHEDULER_ADDRESS (e.g. localhost:8081)

Workers run from dendra-job-workers⁠ with matching NATS and API configuration.

⁠Build and check

Run make check before opening a pull request. Docker builds run the same checks during the image build.

MCP server setup for Cursor: see packages/go/mcp/MCP.md⁠.

⁠Configuration

⁠.env

Copy from sample.env. Loaded via godotenv in both server main packages. Includes:

• Auth: AUTH_PROVIDER (auth0 or keycloak) and provider-specific settings
• Connectors: MONGO_CONNECTORS, INFLUX_CONNECTORS, MINIO_CONNECTORS and per-connector URIs/credentials
• Per-service bindings: SVC_*_CONNECTOR, SVC_* bucket/prefix settings
• API ↔ scheduler: JOB_SCHEDULER_URL, API_SERVER_URL, API key hashes
• NATS: NATS_SERVERS
• Release 2 bridge: WEB_API_V2_* (for dataintegration v2 file import)

Listen addresses: API_SERVER_ADDRESS (defaults to :8080; sample.env uses localhost:8080). JOB_SCHEDULER_ADDRESS (defaults to :8080; sample.env uses localhost:8081 to avoid clashing with the API server).

RPC handlers are mounted under /rpc/ (prefix stripped by the mux).

⁠api.toml

Passed with -config (default api.toml). Defines integration types, shards, and table-discovery rules consumed by data-integration v3 (packages/go/api/config).

⁠jobs.toml

Path set by JOB_PROVIDER_TOMLFILE_PATH on the job scheduler. Preconfigured job specs for the TOML file job provider.

⁠API contracts

Schemas live in proto/api and proto/job, published to the Buf Schema Registry⁠ as dendrascience/api and dendrascience/job. Module overviews: proto/api/README.md, proto/job/README.md. Domain design and handler conventions: docs/⁠ (GO_API_HANDLERS.md⁠, checklist, monitoring).

After changing .proto files, run (from the repo root):

buf lint
buf build
buf breaking --against '.git#branch=main'
buf generate

• buf lint — style and API design rules (STANDARD in buf.yaml).
• buf build — compile protos into a Buf image (resolve imports, including googleapis). Catches schema errors before code generation or publish.
• buf breaking — compare against main for breaking changes (FILE rules).
• buf generate — write Go and Connect stubs to release/go/.

Commit regenerated files under release/go/. Services import release/go/... and generated *connect packages.

This repo is the source of truth for the BSR modules dendrascience/api and dendrascience/job. After the checks above, publish schema updates:

buf push

buf push uploads the compiled modules to the Buf Schema Registry⁠ so other repos (Main App, job workers, generated clients) can consume new commits. Requires BSR authentication (buf registry login). Run from the repo root; both modules in buf.yaml are published.

⁠Store initialization

The API server supports one-off init tasks via flags (then exits):

./output/dendra-api-server -init-action=<action> -init-layer=<layer> -init-store=<store>

Example:

./output/dendra-api-server -init-action=createindexes -init-layer=metadatav3alpha1 -init-store=mongo

Layers for Mongo today: authv3alpha1, metadatav3alpha1, filev3alpha1, dataintegrationv3alpha1, monitoringv3alpha1. Environment variables follow {ACTION}_{STORE}_{LAYER}_* (see CREATEINDEXES_*, SETUP_* in sample.env).

Schema changes are applied through these init actions. Automated migrations are not yet implemented (datalayer.InitActions).

⁠Docker

Images are built from docker/:

Builds run make inside the image. Images do not include MongoDB, NATS, Influx, or MinIO—configure those separately per sample.env.

⁠Development

• Workspace: go.work links lib/go, packages/go/api, packages/go/job-scheduler, and packages/go/mcp.
• Quality: make check runs formatting, staticcheck, and vet. staticcheck.conf sets checks = ["inherit"].
• New RPC service: add .proto definitions, run buf generate, implement handlers under packages/go/api/services/, register in the relevant Setup and main.go.
• New datalayer store: implement the domain interface under datalayer/<store>/, register a connector in conns/, and add a store branch in the service module’s Setup using SVC_*_CONNECTOR.

⁠Testing

In-repo unit tests are limited; run them with make test-api, make test-job-scheduler, and make test-mcp.

Validation that spans NATS, databases, IdP, and Release 2 bridges depends on those services running. Cross-service and end-to-end checks are intended to live in separate test tooling repositories outside this codebase, so core services stay focused while integration scenarios remain testable.

⁠Further reading