Sign inSign up

fors33/data-bridge

By fors33

•Updated about 1 month ago

Config-driven ingestion and normalization for file, batch, and live connectors.

Image
Developer tools
Data science
Monitoring & observability
0

1.6K

fors33/data-bridge repository overview

⁠Fors33 T3thr

CI Release Docker Tag Docker Pulls License

T3thr is a config-driven tool for processing time-series data from local files (free tier) or live connectors (subscription-gated), producing clean outputs for downstream analysis.

⁠Limitation of Liability

T3thr is a deterministic ingestion engine provided "AS IS". Fors33 is not liable for data dropped due to network latency, third-party API rate limits, or improper local hardware configuration. The operator assumes all responsibility for regulatory compliance, data retention, and hardware provisioning. For the full EULA, see fors33.com/products/t3thr.

⁠Quick start (Docker)

  • Mount your config directory to /app/config
  • Default config path inside the container is /app/config/default.toml
⁠Recommended hardened runtime flags

If you run with --read-only, you must also:

  • mount /app/config as read-only
  • mount a writable destination for outputs (from BridgeConfig.output, e.g. /app/out)

Example (Mac / Linux):

  • --cap-drop=ALL
  • --security-opt no-new-privileges:true
  • --read-only
⁠Mac / Linux
docker run --rm \
  --cap-drop=ALL \
  --security-opt no-new-privileges:true \
  --read-only \
  -v "$(pwd)/config:/app/config:ro" \
  -v "$(pwd)/data:/app/out" \
  fors33/data-bridge \
  --config /app/config/default.toml
⁠Windows (PowerShell)
docker run --rm `
  --cap-drop=ALL `
  --security-opt no-new-privileges:true `
  --read-only `
  -v ${PWD}/config:/app/config:ro `
  -v ${PWD}/data:/app/out `
  fors33/data-bridge `
  --config /app/config/default.toml
⁠GitHub Container Registry (Alternative)

You can also pull from GitHub Container Registry:

docker pull ghcr.io/fors33-official/data-bridge:0.6.1
docker pull ghcr.io/fors33-official/data-bridge:latest

⁠Live connectors (subscription required)

Live connectors (WebSocket / REST / Message Bus / gRPC / Syslog / UDP Raw / CDC) require FORS33_LICENSE_KEY.

docker run --rm \
  -e FORS33_LICENSE_KEY="your_key" \
  -v "$(pwd)/config:/app/config" \
  fors33/data-bridge \
  --config /app/config/default.toml
⁠CLI overview
  • Legacy style (unchanged): t3thr --config /app/config/default.toml (equivalent to t3thr run --config …).
  • Explicit subcommands: t3thr run …, t3thr generate --connector <slug> (templates under config/templates/), t3thr validate --config …, t3thr migrate …, t3thr wizard (same as the config_wizard binary).
  • Thin wrappers: validate_config, migrate_config binaries delegate to t3thr (stable for scripting).
Release notes (expand)
0.6.1 – L3dgr kernel parity patch
  • File connector: MappedAcceptedWriter, frictionless ingest without field_map, configurable CSV delimiter, follow_links(false) in batch walks.
  • Live connectors: Kafka manual commit after enqueue; syslog/UDP UTF-8 dead-letter; gRPC reconnect with backoff; MySQL CDC resume_gtid.
  • CLI: brand-correct t3thr --help about string.
  • Semver synced in Cargo.toml and pkg/Dockerfile label at 0.6.1.
  • See docs/bridge-release-notes-0.6.1.md⁠.
0.6.0 – Transport parity and batch caps
  • Batch execution caps: [execution] table (max_records, max_duration_sec, max_pages) with [FORS33] batch complete stderr signaling; caps apply to CSV/file batch loops and live connectors.
  • Multi-feed outputs: accepted_path_by_feed, accepted_prefix_by_feed, channel_scoped_accepted for per-channel accepted files.
  • WebSocket: binance_futures, binance_ws_api, HMAC signing, multi-channel Kraken; optional feed on accepted JSONL rows.
  • Postgres CDC: wal2json and pgoutput slot plugins; optional resume_lsn.
  • Build and release: native_certs feature gate for slim builds; CI validates Docker image before registry push; bundled config/default.toml and examples/sample_input.csv.
  • Onboarding: TERMINOLOGY.md; Docker Hub README sync on release workflow.
  • Semver synced in Cargo.toml and pkg/Dockerfile label at 0.6.0.
  • See docs/bridge-release-notes-0.6.0.md⁠.
0.5.0 – Extension Rust bridge parity (standalone)

Summary: Standalone Rust stack matches the L3dgr embedded T3thr transport layer for TLS observability ([T3thr:CONNECTION_META]), rustls-centric connectors, FORS33_SECRET_* placeholder expansion (additive next to T3THR_* env tables), nested Kafka/MQTT config, t3thr generate from embedded config/templates/, state file locking, Tokio block_in_place for file ingestion, unified subcommand CLI with backward-compatible bare --config, --validate-only, --config-wizard, and retained validate_config / migrate_config binaries. Published images now include config_wizard, validate_config, and migrate_config under /usr/local/bin/.

Technical note: config crate merges FORS33_SECRET with __ separators after loading the file (see docs/bridge-release-notes-0.5.0.md⁠).

0.4.0 – wizard, connectors, batch
  • Frictionless configuration: Wizard and templates for standard providers (Kraken, Binance, Alchemy, Infura, Postgres, MySQL, and more).
  • Interactive wizard: run cargo run --bin config_wizard, t3thr wizard, or t3thr --config-wizard.
  • Pre-configured examples: config/ directory (sample TOMLs); canonical generator templates live under config/templates/ from 0.5.0 onward.
  • Connectors: Syslog (RFC 5424/3164), UDP JSON, CDC (Postgres/MySQL), Kafka/MQTT, gRPC.
  • Batch mode: mode = "batch"; state file .t3thr-state.json with --reset-state / --no-state.
  • Registries: Docker Hub fors33/data-bridge and ghcr.io/fors33-official/data-bridge.
0.3.0 – secrets in the environment, not on disk

Preferred (direct mapping): optional env_* tables map a wire key to an environment variable name (T3THR_[A-Z0-9_]+). Values are sent as-is (put a full Bearer … in the env var if needed).

  • [connector.rest.env_headers]
  • [connector.websocket.env_headers]
  • [connector.grpc.env_metadata]
  • [connector.message_bus.kafka.env_client_properties] / [connector.message_bus.mqtt.env_client_properties] (nested; Kafka/MQTT also support literal client_properties maps)

Deprecated path: whole-value ${T3THR_*} in literal maps still resolves with [DEPRECATION] warning.

License key PEM: optional FORS33_RUNTIME_PUBKEY_PEM for local Ed25519 verification before embedded issuer PEM.

Validate: validate_config path/to.toml or t3thr validate --config … or t3thr --validate-only --config ….

The default slim build supports file, CSV, and REST; full_engine adds WebSocket, message bus, gRPC, syslog, UDP raw, CDC, Parquet (the fors33/data-bridge image builds with full_engine).

⁠Documentation

Tag summary

Content type

Image

Digest

sha256:2276047d1…

Size

41.1 MB

Last updated

about 1 month ago

docker pull fors33/data-bridge