iliasaz/mlx-embeddings-server repository overview

⁠mlx-embeddings-server

An OpenAI-compatible embeddings server that runs nomic-embed-text-v1.5 on a Linux CPU, in one container, with no Python, no GPU and no network at run time. It is a single Swift binary on MLX⁠ — Apple's array framework — using its CPU backend, which is the only one a plain Linux host has.

Source, full documentation and the patches that make MLX build on Linux: https://github.com/iliasaz/mlx-embeddings-server⁠

⁠Quick start

docker run --rm -p 8080:8080 iliasaz/mlx-embeddings-server

curl -s localhost:8080/v1/embeddings \
  -H 'content-type: application/json' \
  -d '{"model":"nomic-embed-text-v1.5","input":"the cat sat on the mat"}'

It speaks the OpenAI embeddings API, so the official SDKs work unchanged:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
client.embeddings.create(model="nomic-embed-text-v1.5", input=["hello", "world"])

Nomic's task prefixes are applied for you. Ask for one with input_type (search_document, search_query, classification, clustering) or with a model-name suffix such as nomic-embed-text-v1.5/search_query, which works through clients that have nowhere to put an extra field. Matryoshka truncation is supported via dimensions (1–768), and encoding_format accepts both float and base64.

⁠Tags

The model weights are 548 MB of the full image — 58% of it — so :slim is the tag to pull if you already have them. The rest is the Ubuntu 24.04 rootfs and the Swift 6.3.3 runtime (260 MB together), the BLAS/LAPACK stack, and a 47.8 MB server binary.

docker run --rm -p 8080:8080 \
  -v "$PWD/nomic-embed-text-v1.5:/opt/models/nomic-embed-text-v1.5:ro" \
  iliasaz/mlx-embeddings-server:slim

⁠Is it the same model?

Yes, and it was measured rather than assumed. Against an independent implementation of the same weights (jkrukowski/swift-embeddings on CoreML/MLTensor, macOS), over a 30-text fixture:

• minimum per-text cosine 0.9999999999980043, 0 of 30 below the pre-committed 0.999
• identical ordered top-3 documents on 6 of 6 queries

The method, the negative controls and the reproduction script are in docs/VERIFICATION.md⁠.

⁠Performance, honestly

One CPU core is not a GPU. On an M5 Max under Docker Desktop, linux/arm64: about 11.4 documents/s (≈1227 tokens/s) for ~100-token documents, 0.445 s from docker run to the first 200 on /health. Batching buys nothing — the cost is linear in tokens — and requests serialise on the model, so scale by running more containers, not more connections.

⁠Configuration

Every flag has an environment variable, which is how the image is configured: MODEL_PATH, MODEL_NAME, MODEL_ALIASES, HOST, PORT, LOG_LEVEL, API_KEY, MAX_BATCH_SIZE, MAX_BATCH_TOKENS, MAX_SEQUENCE_LENGTH, MAX_REQUEST_BYTES, DEFAULT_INPUT_TYPE, TRUNCATE_LONG_INPUTS, CORS_ALLOW_ORIGIN. An explicit flag always wins. docker run --rm iliasaz/mlx-embeddings-server --help prints the full list.

The server binds 0.0.0.0 inside the container and has no authentication by default. To reach it only from the host, publish it on the loopback interface — docker run -p 127.0.0.1:8080:8080 — or set --api-key.

⁠Licenses

The server's own code is MIT. The image also redistributes MIT-licensed forks of mlx-swift and mlx-swift-lm, and the latest/0.1.1 tags redistribute the Apache-2.0 nomic-ai/nomic-embed-text-v1.5 weights, pinned at revision e9b6763. LICENSE and NOTICE.md ship inside the image at /usr/share/doc/mlx-embeddings-server/.