hectorcardoso/malachi repository overview

⁠Malachi

An open-source, 100% Elixir reimplementation of LinkedIn's NorthGuard log-storage architecture: a CP, horizontally scalable log broker.

Clients speak topics, keys and opaque cursors, never partitions or offsets. That indirection is the point: the broker can split a topic's storage and restripe it across nodes underneath a running client without breaking it, because the client never held a coordinate that could go stale. The control plane is replicated by quorum (Raft, via ra).

Documentation⁠ · Source⁠ · Benchmarks and chaos results⁠

⁠Introduction

⁠Quick start

docker run -d --name malachi \
  -p 127.0.0.1:4040:4040 -p 127.0.0.1:4041:4041 \
  -e MALACHI_ADMIN_PASS="a-long-local-only-password" \
  -e MALACHI_REQUIRE_TLS=false \
  hectorcardoso/malachi:latest

Then open the dashboard at http://localhost:4041⁠, and check http://localhost:4041/health, which should answer with an ok status.

Three things in that command are load-bearing, so it is worth saying why rather than leaving you to find out from a crash loop.

The image runs a production release, and a production release refuses to start on a weak or default password (admin123 and friends are rejected outright, and anything under 12 characters is too short). It also requires TLS unless told otherwise, which is why MALACHI_REQUIRE_TLS=false is there.

And the ports are bound to 127.0.0.1 rather than published on every interface, because the other two choices make that one matter: with TLS off and a password copied from a public page, -p 4040:4040 would hand a plaintext broker to anyone who can route to your host. The loopback prefix is what keeps this a local example. Do not delete it to reach the container from another machine; make the deployment a real one instead.

⁠Running it for real

Drop MALACHI_REQUIRE_TLS=false, mount certificates, and point at their container paths. Host paths in those variables resolve inside the container, where they do not exist, so a bind mount is not optional:

docker run -d --name malachi \
  -p 4040:4040 -p 4041:4041 \
  -v /etc/malachi/certs:/certs:ro \
  -e MALACHI_TLS_CERTFILE=/certs/server.pem \
  -e MALACHI_TLS_KEYFILE=/certs/server-key.pem \
  -e MALACHI_ADMIN_PASS="$(openssl rand -base64 32)" \
  hectorcardoso/malachi:latest

⁠Ports

⁠Tags

Built for linux/amd64 and linux/arm64, so Apple Silicon and Graviton pull a native image rather than emulating one.

⁠Persistence

Nothing survives the container being recreated unless you mount a volume and point the two data directories at it. Left unset, they land under /tmp inside the container's writable layer, which is where the benchmark setups deliberately put them.

The distinction matters more than it looks. A docker restart keeps that writable layer, so data written under /tmp is still there afterward and the setup appears to persist. It is docker rm followed by a fresh run, or a docker compose up that recreates the container, that takes it, and that is the operation an upgrade performs.

docker run -d --name malachi \
  -p 127.0.0.1:4040:4040 -p 127.0.0.1:4041:4041 \
  -v malachi-data:/app/data \
  -e MALACHI_LOG_DATA_DIR=/app/data/log \
  -e MALACHI_RA_DATA_DIR=/app/data/ra \
  -e MALACHI_ADMIN_PASS="a-long-local-only-password" \
  -e MALACHI_REQUIRE_TLS=false \
  hectorcardoso/malachi:latest

Both directories matter, and for different reasons. MALACHI_LOG_DATA_DIR holds the log segments, the records themselves. MALACHI_RA_DATA_DIR holds the replicated control plane: topics, ranges, and the user accounts, ACLs and lockouts. Persist only the first and the data survives while the users who could read it do not.

Mount at /app/data specifically. A named volume is created owned by root, and Docker only hands it the mount point's ownership when that directory already exists in the image. /app/data does; a fresh /data would leave the broker unable to write into its own volume.

⁠Configuration

The full list is in the documentation⁠. These are the ones worth knowing before the first run:

⁠Watching it work

Malachi exports both halves of its own observability natively, and the repository ships a compose stack that wires them up for you:

git clone https://github.com/HectorIFC/malachi.git && cd malachi
docker compose up -d

That gives you the broker plus Jaeger on port 16686 and Prometheus on 9090. A single produce shows up in Jaeger as a distributed trace, malachi.produce at the root with the broker's append and the quorum commit nested under it, and Prometheus carries around thirty malachi_ series covering throughput, authentication, replication and the integrity scrub.

For three nodes at replication factor 3 on durable volumes, with the same two:

docker compose -f docker-compose.cluster-durable.yml up -d

Both stacks are local examples in the same sense as the quick start above, and for the same reasons: they carry placeholder credentials that anyone can read in the repository, they set MALACHI_REQUIRE_TLS=false, and Jaeger and Prometheus have no authentication at all. Every port they publish is therefore bound to 127.0.0.1. Override the credentials through the environment before either one goes anywhere:

MALACHI_ADMIN_PASS="$(openssl rand -base64 32)" \
MALACHI_PRODUCER_PASS="$(openssl rand -base64 32)" \
  docker compose up -d

Tracing is off by default in every other setup, and deliberately: the sampler drops every span, so the instrumentation on the produce path costs a function call that declines to record. Turn it on when you need it, and use MALACHI_TRACING_SAMPLE_RATIO on anything busy.

⁠Metrics

GET /metrics on the dashboard port content-negotiates: ask for text/plain and you get the Prometheus text exposition, ask for anything else and you get the dashboard's JSON. It requires an authenticated user, and Malachi sessions expire, so a scraper needs a token that gets refreshed rather than one pasted into a config. The repository's compose stack includes a small sidecar that does exactly that, and it is the shortest working example to copy.

⁠Health

livenessProbe:  { httpGet: { path: /health, port: 4041 } }
readinessProbe: { httpGet: { path: /ready,  port: 4041 } }

Note that /metrics is authenticated and is not a health probe.

⁠License

MIT. Source, issues and the full documentation live at github.com/HectorIFC/malachi⁠.