teyfix/redpanda-connect repository overview

⁠teyfix/redpanda-connect

Redpanda Connect with CUE rendering, native streams reload and Process Compose supervision. See the shared pipeline development contract and examples⁠.

⁠Pipeline development images

teyfix/vector and teyfix/redpanda-connect package their respective engine, CUE, the shared bootstrap, Process Compose, gomplate and watchexec. Connect builds pinned upstream source with the JetStream patch described below; Vector retains its official Alpine runtime. Each is self-contained; no broker or application binary is bundled. Versions live in docker-bake.hcl.

Mount /config/dev.yaml and your source tree at /src:

source: /src
watch: [/src, /shared-config]
render:
  command: [cue, export, ., -e, files, --out, json]
  timeout_seconds: 60
provision:
  command: [/src/provision.sh]
  timeout_seconds: 120

source, the default render command, and /src watching are defaults. Provisioning defaults to /bin/true. Commands are argv lists, executed from source; they inherit the container environment. An optional consumer-owned provisioner runs once per startup after valid rendering and before the engine. Give a resource exactly one initialization owner. Include required clients in a consumer extension image when a provisioning command actually needs them.

The render command writes a JSON object to stdout mapping relative filenames to configuration objects. The usual CUE expression is files. Send diagnostics to stderr. Examples: Vector⁠, Connect⁠, configuration⁠. Custom render commands, including cue cmd, must honor the same stdout contract. Arguments preserve spaces and literal shell characters; use an explicit shell command only when you want shell interpretation.

Vector files are flat name.yaml fragments. Connect uses streams/name.yaml for pipelines and optional root.yaml for service-wide logger, metrics and resources. Names contain letters, digits, underscore or hyphen. All generated files come from the current render; omitted files are removed on activation. Other formats, arbitrary filesystem writes and native configuration mounts are outside this renderer contract. Native configurations may refer to mounted consumer mappings; include all their dependencies in watch.

Every edit triggers a queued, debounced render in a new staging directory. CUE validates the output boundary; vector validate --no-environment or Connect lint validates native configuration before activation. These are configuration checks, not proof that remote sinks are reachable. A failed edit logs its error, sets /run/teyfix-pipeline/rejected, and retains the last valid generation. Readiness remains tied to the serving engine, so a rejected edit does not take a working pipeline offline. The marker clears on the next successful render. Initial render or provisioning failure exits the container.

Activation swaps a symlink to the complete validated generation. Vector uses its native polling config watcher; Connect uses its native streams watcher. After the swap, renaming retired stream files makes Connect detach its old inode watches even when a file is still held open; the retired generation is then removed. Native reload resolves the stable paths, including added/removed streams. Connect root configuration changes restart only the engine through Process Compose, because its streams watcher does not reload the root. Source/config mounts must be directories for atomic editor replacements to be observable. Changing /config/dev.yaml requires a container restart; watch all other imported files, packages and mapping directories explicitly.

The local readiness API is reserved: Vector uses 127.0.0.1:8686/health, Connect uses 127.0.0.1:4195/ready. Vector's teyfix-runtime.yaml, api and data_dir are image-owned; do not declare them in fragments. Connect root HTTP binding and JSON logger format are image-owned. Set http.bind: 0.0.0.0 in dev.yaml to expose Connect metrics/readiness on port 4195 to Prometheus on the project network (or Vector API on port 8686). The default remains 127.0.0.1; ports stay fixed for native readiness. For Connect, configure root.yaml metrics with prometheus: {}. Connect's remote stream-management API is not enabled; application HTTP inputs can bind their own configured addresses. An engine crash or watcher exit is visible through Process Compose; shutdown stops the watcher before the engine.

Declare shared Connect processor, rate-limit, cache, input and output resources in the native *_resources fields of root.yaml. The image separates them into a private generated resources.yaml, passes that file to both native validation and runtime --resources, and keeps observability configuration separate. Resource-only changes restart the engine just like other root changes; stream-only edits retain native reload. Consumers do not emit the internal resource file or manage another launcher.

A watched dependency can change without changing rendered YAML, such as a Connect mapping: from file. After validating such an edit, the image restarts Connect to recreate its native resource and stream parsers. This also applies to watched no-op edits. The watcher starts listening after initial rendering, without running a duplicate startup render.

Mount /data for engine state and /var/log/vector or /var/log/redpanda-connect for native JSON engine logs (override paths.log in dev.yaml), readable by an external collector. User pipelines own their outputs; console outputs share the engine log file and retain their configured format. Do not put generated runtime output in a watched source directory. No Python is installed in runtime images.

See Compose⁠. Start from the project root and substitute your source directories and environment/secret mounts. .state bind mounts survive docker compose down -v; explicitly remove the scoped state after stopping when resetting. Preserve reusable .cache data separately.

Build with docker buildx bake vector redpanda-connect --load. Smoke targets vector-smoke and redpanda-connect-smoke exercise rendering, dependency changes, native reload, stream/fragment removal and addition, invalid edit retention, Connect root restart, bounded provisioning, logs and shutdown. CI runs amd64 and arm64 before publication.

⁠Broker tooling and JetStream delivery

For native topic/stream provisioning, copy the broker-tools artifact⁠ into the consuming pipeline extension. Its rpk and nats binaries use the existing provision.command hook. Resource declarations stay in the consumer; designate one initialization owner.

Connect is built from pinned upstream source with the JetStream acknowledgement-renewal patch⁠, carried from Animatrix and reviewed against 4.114.0. It uses the actual consumer AckWait/backoff to send InProgress while processing, stops renewal at terminal Ack/Nak or disconnect, and retains native redelivery after shutdown/failure. This does not make processing exactly-once. Native tests cover successful long-running work, competing pulls, reconnect, restart and negative acknowledgements against a real JetStream server. The binary reports the -teyfix suffix; upstream connector licenses remain installed under /usr/local/share/licenses/redpanda-connect.

Maintained from the source documentation⁠. See published tags⁠ and the service integration guide⁠.