jylus/jylus-migrate repository overview

⁠Jylus Migrate

Assess Chroma Cloud, approve a record copy, verify it and compare retrieval.

Run the ready-built container on your own machine. Source credentials stay local. The default assessment makes no Jylus writes and needs no Jylus API key.

Docker Hub⁠ · Website guide⁠ · Query comparison⁠ · Qualification and limits⁠

This is a preview record-copy evaluation, not a drop-in Chroma vector-search replacement. Original embeddings remain in the private local export. External dense-vector indexing, equivalent similarity metrics, fork lineage and specialised hybrid search are not supported. Hosted Chroma Cloud acceptance is still open.

⁠1. Configure local credentials

Install Docker with Linux containers. This release is qualified for Linux amd64; ARM and other SDK versions are not certified. No source download or local build is needed. Create secrets/ and work/ in a new local directory.

Save secrets/chroma.json using your own Chroma Cloud credentials:

{"api_key":"YOUR_CHROMA_CLOUD_KEY","tenant":"YOUR_TENANT","database":"YOUR_DATABASE","region":"us"}

Use eu for the supported European endpoint. Tenant/database may be omitted when CloudClient can infer them. Use a source key permitted to read the collection. The tool never creates, updates, deletes or forks your source collection.

Create migrate.env with paths only:

CHROMA_CREDENTIALS_FILE=/run/secrets/chroma.json
JYLUS_API_KEY_FILE=/run/secrets/jylus.key

On Linux/macOS, set chmod 600 secrets/chroma.json. Keep keys, exports and the work directory out of source control. Do not create jylus.key for assessment. Only approved transfers need that separate key with events:read and events:write for a Jylus evaluation workspace.

⁠2. Discover and assess

macOS / Linux, after creating the secret files:

mkdir -p work
docker run --rm --platform linux/amd64 --pull missing --read-only --memory=768m --cpus=1 --pids-limit=128 \
  --cap-drop=ALL --security-opt=no-new-privileges --tmpfs /tmp:rw,size=128m \
  --user "$(id -u):$(id -g)" --env-file ./migrate.env \
  --mount "type=bind,src=$PWD/secrets,dst=/run/secrets,readonly" \
  --mount "type=bind,src=$PWD/work,dst=/work" \
  docker.io/jylus/jylus-migrate:0.1.1 chroma-cloud

PowerShell, after creating the same files:

New-Item -ItemType Directory -Force -Path work | Out-Null
docker run --rm --platform linux/amd64 --pull missing --read-only --memory=768m --cpus=1 --pids-limit=128 `
  --cap-drop=ALL --security-opt=no-new-privileges --tmpfs /tmp:rw,size=128m `
  --env-file ./migrate.env `
  --mount "type=bind,src=$($PWD.Path)/secrets,dst=/run/secrets,readonly" `
  --mount "type=bind,src=$($PWD.Path)/work,dst=/work" `
  docker.io/jylus/jylus-migrate:0.1.1 chroma-cloud

Docker pulls the published image automatically. The command discovers up to 1,000 collections and assesses the first 20, with coverage reported. Add --collection "your-collection" to assess a particular collection. You receive compatibility findings, sampled storage estimates and a reviewable assessment. Estimated bytes are not a bill or an index/RAM/replication capacity guarantee.

⁠3. Approve a copy

Keep the application on its existing system. Select an intentionally frozen evaluation collection and review its assessment. Create secrets/jylus.key with your separate evaluation-workspace key; protect it like the source key. Choose the workspace retention and review its storage cap before transferring.

Append these arguments to the same Docker command, after chroma-cloud:

--collection your-frozen-collection --execute --approve REVIEWED_SHA256 --record-storage-only --frozen-source

Only explicit approval enables Jylus writes. Compatible text, original IDs, metadata and provenance go through the public /api/v1/events endpoint, at most 100 events and 900,000 encoded bytes per batch. Original vectors remain local. The tool does not activate extra charges, change your plan or switch traffic.

⁠4. Verify, resume and compare

Progress reports verified records. A checkpoint commits only after a durable receipt, matching integrity digest and exact record readback. Ctrl+C preserves progress. Resume the same approved command with --resume, keeping the same work directory, target key and batch size. Stable identifiers and idempotency keys prevent another logical copy within the same job.

Use --queries /work/queries.json for representative original-ID and retrieval comparisons; the query comparison section below provides the format. Chroma receives saved source query vectors; Jylus receives the original question as text. No embedding/model provider is called. Different engines can return different rankings. Supplied independent relevance judgments are optional, and missing judgments remain unknown. A successful record copy is not vector parity.

Count verification requires a complete native aggregation, with a 50,000-record scan bound. Partial counts stay explicitly unverified. The source remains untouched; reports never automatically approve application cutover.

For the interruption acceptance check, append --stop-after-batches 1 to the approved command; exit 75 is expected. Repeat without that option, adding --resume, and confirm counts and integrity. A fresh directory starts a new evaluation, not a retry of the existing job.

⁠Support and compatibility

Only Chroma Cloud is currently implemented. Pinecone, Qdrant, pgvector and Elasticsearch are planned investigations after hosted Chroma acceptance passes. They are not advertised as available connectors.

See the qualification section below for the 69-test Linux qualification, official SDK HTTP-fixture limits and 1,001-record real-Jylus interruption/resume evaluation. Those results do not certify hosted Chroma Cloud or vector-search equivalence.

Report issues at this repository⁠, omitting keys and customer records. Security concerns: contact [email protected]⁠ privately. Do not post keys or customer data.

The Apache 2.0 licence covers this migration client only. Required Qdrant attribution is in /app/NOTICE inside the image. The full licence is at /app/LICENSE. The proprietary Jylus engine is not included or licensed by this repository or container.

⁠Query comparison

Save representative cases in work/queries.json, then append --queries /work/queries.json to the approved execution/resume command:

[
  {"name":"original-record","ids":["YOUR_ORIGINAL_ID"]},
  {"name":"shipment-delay","text":"Why is this shipment delayed?",
   "query_embedding":[0.1,0.2],"k":10,
   "where":{"order_id":{"$eq":"ORD-1001"}}}
]

The two-dimensional vector above is illustrative. Supply the actual saved Chroma query vector with the collection's full dimension; the tool does not generate embeddings. Exact-ID cases need no query vector. Chroma receives the saved vector, while Jylus receives the original text. This compares the same question through different engines, not identical vector-search behaviour.

⁠Qualification and limits

⁠Version 0.1.1 preview

This utility does not change the proprietary Jylus engine. The reference approach is pinned in NOTICE, the Python base image by digest, and all 35 third-party dependency wheels by version and SHA-256 for Linux amd64 / CPython 3.12.

The expanded suite contains 69 tests. Linux runs all cases; Windows reserves the native symlink test for Linux rather than assuming administrator privileges. Immutable image and final execution receipts are recorded in the repository's release notes. Repeats are not additional unique tests.

The suite tests official CloudClient token authentication, discovery, original IDs/Unicode/whitespace/metadata, dense-vector export, configuration capture, source and target rate limits, source deadlines, credential separation, review gates, deterministic target IDs, durable receipt/digest checks, interrupted and unknown-outcome writes, resume, target identity binding, corrupt checkpoints, same-count source mutation, missing text, partial/duplicate readback, unsupported filters, dimension mismatches and honest quality reporting. A generated 2,500-row source exceeds 5 MiB of exported JSON while traced exporter Python heap stays below 2 MiB; this is an exporter allocation test, not a whole-process RAM bound.

The official CloudClient suite uses a synthetic HTTP fixture, not the hosted Chroma Cloud service. Source fixture networking is restricted to loopback. SDK authentication and request serialization are real; stored vectors are supplied explicitly, and no embedding/model provider is called. An isolated Linux container executes the tests without external networking under 768 MiB / one CPU.

A separate test also used the official source-client fixture with real Jylus public HTTP handlers, receipts, broker and native storage, behind an isolated QA portal/metadata database. It copied three synthetic records, deliberately stopped with exit 75, resumed with exit 0, verified every original record and the namespace count, and completed exact-ID/text-retrieval comparisons. The same source credential never entered a Jylus request. There were no customer accounts created in Atlas or serving engine/configuration changes. This isolated metadata database was a standalone Mongo fixture; production billing/accounting transaction behavior is not qualified by that test. SSH test transport adds latency, so those numbers are not production performance benchmarks.

The tenant-restricted QA relay refused an overly broad 50,000-record comparison scan for a three-record collection. Comparison scan bounds now follow the source count (100 minimum, 50,000 maximum); the same-source rerun passed. The earlier local prototype also caught native system enrichment being mistaken for source data mutation. Imported source subtrees are still compared exactly; system fields outside those subtrees are allowed. Failed setup/test receipts were preserved.

Version 0.1.1 repeated real-target interruption/resume with 1,001 synthetic records, batches of 100, and an independent complete native namespace count. No missing or duplicate logical records remained. This larger run exposed an immediate readback returning fewer records after durable receipt verification. The tool now waits for uncached visibility of those same records, verifies all returned content immediately and preserves the checkpoint on a deadline. The same 1,001-record namespace resumed successfully. This is a bounded operational test, not a hosted Cloud or production-scale benchmark.

New regressions cover failed-plus-ungraded quality gates, missing judgments, malformed/duplicate baseline IDs and distances, both engines' judged-pool scores, official multi-field filter serialization, representative source mutations, source-ID lookups, stable transient retries, TLS refusal, Retry-After handling, visibility waits/conflicts/deadlines, aggregation counts and PowerShell UTF-8 BOM.

⁠Gates that remain open

• A representative hosted Chroma Cloud collection has not been run here: no customer Cloud credentials/collection were supplied. Customers should run the documented frozen-collection acceptance and interruption/resume procedure.
• External dense-vector dimensions, metrics, ANN indexing and query compatibility are not established by preserving vector JSON/digests. This version keeps full vectors locally and reports the target vector index as unsupported.
• Chroma fork lineage/copy-on-write and specialised Search API/hybrid expressions are not reproduced. Negative/missing-field filters, nested Boolean/document filters and literal dotted metadata keys are not qualified and are refused.
• Missing original text blocks target transfer; URI assets are not fetched.
• Every expected target ID/content is checked. Independent namespace count is verified only when the bounded native aggregation reports a complete result; partial counts are explicitly unverified. Larger-scale qualification is open.
• ARM, other client versions, million-record throughput, customer relevance, production billing, recovery/failover and application cutover are not certified.
• No QASPER/FinQA/TEMPO benchmark or native ranking formula change occurred.

Reports never approve application traffic switching. Missing relevance judgments remain unknown. Future connectors remain deferred until the Cloud acceptance gate passes. A passing fixture suite is not a claim that hosted Cloud acceptance or vector-search compatibility has passed.

This Docker Hub release was additionally pulled anonymously by its exact manifest digest and passed all 69 Linux client tests. All 17 files under /app matched the qualified migration-client allowlist. Manifest: sha256:d6da9960c32ab89474a07328ea6e55f276b9214e999f8b193d99092db8141630.