eigenlake/eigenlens-chat repository overview

⁠EigenLens

Cluster existing vector embeddings with DBSCAN and extract topic keywords with BERTopic. Use the MCP worker from your own agent, or run EigenLens Chat to explore Qdrant collections with interactive plots.

⁠Public images

Both images support Linux AMD64 and ARM64. Docker chooses the matching platform. No Docker Hub login, GitHub account, repository clone or local build is required.

docker pull eigenlake/eigenlens:0.1.0
docker pull eigenlake/eigenlens-chat:0.1.0

Use version tags or digests for reproducible deployments. Source repositories remain private; these public images and the instructions below are sufficient to run the application.

⁠Start the complete app

You need Docker with Compose and OpenSSL. For a new installation, create a directory and save the following configuration as compose.yaml inside it. This starts three containers: the worker, chat UI and an empty Qdrant database.

# Standalone public Docker Hub deployment for a new installation.
name: eigenlens-demo
services:
  worker:
    image: ${EIGENLENS_IMAGE:-eigenlake/eigenlens:0.1.0}
    command:
    - eigenlens
    - --transport
    - http
    - --host
    - 0.0.0.0
    - --config
    - /app/config/worker.json
    ports:
    - 127.0.0.1:8020:8000
    environment:
      EIGENLENS_MCP_TOKEN: ${EIGENLENS_MCP_TOKEN:?Set EIGENLENS_MCP_TOKEN in .env}
      EIGENLENS_WORKERS: ${EIGENLENS_WORKERS:-1}
      EIGENLENS_TOTAL_WORKING_BYTES: '8589934592'
      EIGENLENS_CPU_BUDGET: '2'
      EIGENLENS_CONNECTION_KEY: ${EIGENLENS_CONNECTION_KEY:-}
      EIGENLENS_DATABASE_TYPE: ${EIGENLENS_DATABASE_TYPE:-}
      EIGENLENS_DATABASE_URL: ${EIGENLENS_DATABASE_URL:-}
      EIGENLENS_DATABASE_API_KEY: ${EIGENLENS_DATABASE_API_KEY:-}
      EIGENLENS_DATABASE_OPTIONS: ${EIGENLENS_DATABASE_OPTIONS:-}
    volumes:
    - worker-data:/data
    init: true
    mem_limit: 8g
    cpus: 2
    restart: unless-stopped
  qdrant:
    image: qdrant/qdrant:v1.15.5
    ports:
    - 127.0.0.1:6333:6333
    volumes:
    - qdrant:/qdrant/storage
    healthcheck:
      test:
      - CMD-SHELL
      - bash -c 'echo > /dev/tcp/127.0.0.1/6333'
      interval: 5s
      timeout: 3s
      retries: 30
  chat:
    image: ${EIGENLENS_CHAT_IMAGE:-eigenlake/eigenlens-chat:0.1.0}
    ports:
    - 127.0.0.1:7860:7860
    environment:
      EIGENLENS_CHAT_HOST: 0.0.0.0
      EIGENLENS_WORKERS: ${EIGENLENS_WORKERS:-1}
      EIGENLENS_CHAT_ALLOW_UNAUTHENTICATED: '1'
      EIGENLENS_CHAT_DATA: /data
      EIGENLENS_CHAT_QDRANT_HOSTS: ${EIGENLENS_CHAT_QDRANT_HOSTS:-localhost,127.0.0.1,::1,qdrant,host.docker.internal}
      EIGENLENS_CHAT_QDRANT_LOCAL_URL: http://qdrant:6333
      EIGENLENS_CHAT_QDRANT_URL: http://localhost:6333
      EIGENLENS_CHAT_USER: ${EIGENLENS_CHAT_USER:-}
      EIGENLENS_CHAT_PASSWORD: ${EIGENLENS_CHAT_PASSWORD:-}
      EIGENLENS_MCP_URL: http://worker:8000/mcp
      EIGENLENS_MCP_TOKEN: ${EIGENLENS_MCP_TOKEN:?Set EIGENLENS_MCP_TOKEN}
      OPENROUTER_API_KEY: ${OPENROUTER_API_KEY:-}
    volumes:
    - chat-data:/data
    - chat-models:/models
    extra_hosts:
    - host.docker.internal:host-gateway
    init: true
    mem_limit: 3g
    cpus: 2
    restart: unless-stopped
    depends_on:
      worker:
        condition: service_healthy
      qdrant:
        condition: service_healthy
volumes:
  worker-data: null
  qdrant: null
  chat-data: null
  chat-models: null

From that directory, generate the runtime settings once and start the services:

if [ ! -f .env ]; then
  (umask 077
   printf 'EIGENLENS_MCP_TOKEN=%s\nEIGENLENS_CONNECTION_KEY=%s\nEIGENLENS_WORKERS=1\n' \
     "$(openssl rand -hex 32)" "$(openssl rand -base64 32 | tr '+/' '-_')" > .env)
fi
docker compose pull
docker compose up -d --wait

Keep .env private and preserve it across restarts. The MCP token authenticates your running server, independently of public image access. The encryption key is needed to read saved database connections after restarting.

Open http://127.0.0.1:7860/⁠:

1. Click New chat.
2. Enter your Qdrant URL and optional API key.
3. Choose an existing indexed collection.
4. Ask show clusters to analyze the full collection, or show clusters in ATM to search for a topic and analyze its matches.
5. Inspect progress, topic terms, representative examples, the cluster map and downloads in the conversation.

A fresh Qdrant instance contains no records. Populate it or connect your existing database. In this stack, http://localhost:6333 in the connection dialog maps to the Qdrant container. A database running on the host computer can use http://host.docker.internal:PORT. For another hostname, add it to EIGENLENS_CHAT_QDRANT_HOSTS in .env, then recreate the chat container.

Topic search in the chat currently supports cosine collections indexed with sentence-transformers/all-MiniLM-L6-v2, with that model recorded in the collection metadata. Full-collection analysis uses stored embeddings without generating a query embedding. The core MCP also supports externally generated query vectors and its installed database adapters.

Command-based chat works without a model API key. Optionally add OPENROUTER_API_KEY to .env and recreate the chat container to enable the fixed Nemotron free-model agent. That mode sends conversation text, tool summaries and requested example text to OpenRouter/NVIDIA; use public demo data with the free provider.

⁠Use only the MCP worker

If your application already has a vector database and agent, run only the worker. Generate .env as above, then:

docker run -d --name eigenlens-worker --init --restart unless-stopped \
  --publish 127.0.0.1:8020:8000 --memory 8g --cpus 2 \
  --env-file .env --env EIGENLENS_CPU_BUDGET=2 \
  --volume eigenlens_worker-data:/data \
  eigenlake/eigenlens:0.1.0

Do not start this extra worker if the complete app above is already using port 8020. Connect your existing MCP client to http://127.0.0.1:8020/mcp⁠ with Authorization: Bearer <EIGENLENS_MCP_TOKEN> from .env.

The worker provides database connection/retrieval, record registration, background analysis, job status/cancellation, summaries, examples and paginated results. Supported database adapters are Qdrant, Pinecone, Milvus, Weaviate and PostgreSQL/pgvector. It works with stored vectors; no embedding model is downloaded by clustering/topic jobs.

For an OpenAI Agents SDK application, install openai-agents==0.22.2 and supply OPENAI_API_KEY, OPENAI_MODEL, and the runtime worker token in that application's environment:

import asyncio
import os

from agents import Agent, Runner
from agents.mcp import MCPServerStreamableHttp

async def main():
    async with MCPServerStreamableHttp(
        name="EigenLens",
        params={
            "url": "http://127.0.0.1:8020/mcp",
            "headers": {"Authorization": f"Bearer {os.environ['EIGENLENS_MCP_TOKEN']}"},
            "timeout": 300,
            "sse_read_timeout": 300,
        },
        cache_tools_list=True,
        client_session_timeout_seconds=300,
    ) as server:
        agent = Agent(
            name="My assistant",
            model=os.environ["OPENAI_MODEL"],
            instructions="Use EigenLens tools and report actual tool results.",
            mcp_servers=[server],
        )
        result = await Runner.run(agent, "Inspect get_capabilities and explain the available analysis tools and limits.")
        print(result.final_output)

if __name__ == "__main__":
    asyncio.run(main())

Attach the connected server to your existing agent's mcp_servers to retain its other tools. Configure a database through connect_database in application code, keeping credentials out of model prompts, or set the server's EIGENLENS_DATABASE_TYPE, EIGENLENS_DATABASE_URL and optional API key/options before startup. Caller local can then use connect_database without arguments.

The clustering workflow is retrieve_records (or register_records) → run_analysis → get_job → get_summary / get_examples / get_result_page. Use dbscan_search for clustering and bertopic with clusters_from for topics. Application code should poll long jobs and show get_job.info for detailed stages. Follow all result pages to export every assignment or plot every coordinate.

⁠Runtime settings and persistence

Default concurrency is 1 worker, configurable from 1–16 using EIGENLENS_WORKERS in .env. More workers allow simultaneous jobs; they do not split one analysis. The worker defaults to a 20,000-record input limit and a 30-minute job budget. It rejects selections exceeding its limits rather than silently sampling.

The complete app uses up to two CPUs and 8 GiB for the worker and two CPUs and 3 GiB for chat. Topic search may download MiniLM on first use; the model cache persists. Named volumes retain Qdrant records, worker artifacts, conversations and models across restarts.

docker compose stop                 # Stop, keeping data
# Start again from the same directory with the same .env:
docker compose up -d --wait

The standalone configuration uses the eigenlens-demo project and is intended for a fresh installation. Existing two-project installations should retain their original volumes and configuration when changing image references. Do not remove data volumes if you need the saved records or conversations.

Ports are bound to localhost. This setup is intended for a local operator or trusted team; a remote deployment needs HTTPS, authentication and appropriate network access controls.