otakulabz/supabase-chunker

•Updated 5 months ago

Image

870

otakulabz/supabase-chunker repository overview

⁠otakulabz/supabase-chunker

A stateless Deno microservice that handles heading-aware document chunking and batch embedding for Supabase RAG pipelines. Built to run outside Supabase Edge Functions, bypassing the Free-tier 200ms CPU cap that breaks large document processing. Uses text-embedding-3-small (1536 dims) by default — the same model used for query embeddings — so vector dimension mismatches are impossible.

Stack: Deno · Alpine · 54.9 MB · Port 8080 · Runs as non-root deno user

⁠🔌 Endpoints

Method	Path	Description
`GET`	`/health`	Liveness check — returns current rate limiter usage
`POST`	`/chunk-source`	Chunk and embed a document into Supabase

All /chunk-source requests require:

Authorization: Bearer ${CHUNKER_SERVICE_AUTH}

⁠📥 Input Shapes

All four shapes are accepted on the same /chunk-source endpoint. Provide exactly one input field per request.

⁠1. Pull from DB (original flow)

{ "source_id": "f0945db8-cf9b-4798-bbca-9ee5ed7f8fc6" }

⁠2. Raw Markdown Injection

{
  "markdown": "# My Document\n\nContent here...",
  "title": "My Document",
  "source_type": "document",
  "user_id": "optional",
  "project_id": "optional"
}

⁠3. Local File (parsed via Docuflow)

{
  "file_path": "/data/uploads/report.pdf",
  "title": "My Report"
}

⁠4. Remote URL (parsed via Docuflow)

{
  "url": "https://example.com/some-doc.pdf",
  "title": "Optional override title"
}

⁠Agent Metadata (optional on all shapes)

{
  "tagged_by": ["ASAGI", "Rias Gremory"],
  "topics": ["health", "productivity"],
  "agent_notebook": "ASAGI"
}

tagged_by, topics, and agent_notebook are stored in each chunk's metadata column and in the source row so they can be filtered at query time.

⁠⚙️ Required Environment Variables

SUPABASE_URL=https://<ref>.supabase.co
SUPABASE_SERVICE_ROLE_KEY=sb_secret_...
CHUNKER_SERVICE_AUTH=<random-32-byte-hex>
EMBEDDING_API_KEY=...        # or VOIDAI_API_KEY / OPENAI_API_KEY

Generate a secure auth secret:

openssl rand -hex 32

⁠🔧 Optional Environment Variables

Variable	Default	Notes
`EMBEDDING_BASE_URL`	`https://api.voidai.app/v1`	Any OpenAI-compatible `/embeddings` endpoint
`EMBEDDING_MODEL`	`text-embedding-3-small`	1536-dim model
`EMBED_BATCH_SIZE`	`128`	Inputs per embedding API request
`EMBED_CONCURRENCY`	`4`	Batches in flight at once
`EMBED_MAX_RETRIES`	`3`	Retries on 5xx / 429 with 1s / 3s / 9s backoff
`EMBED_RATE_LIMIT`	`100`	Max embed API requests per rate window
`EMBED_RATE_WINDOW_MS`	`60000`	Rolling window in ms (default = 1 min)
`INSERT_BATCH_SIZE`	`32`	Rows per Supabase insert statement
`RAG_SCHEMA`	`public`	Postgres schema for RAG tables
`RAG_TABLE`	`neuro_rag`	Source metadata table
`RAG_CHUNKS_TABLE`	`neuro_rag_chunks`	Chunk + embedding table
`APP_SCHEMA`	`notebooklm-demo`	Set `""` to disable dual-write entirely
`APP_SOURCES_TABLE`	`notebook_sources`
`DOCUFLOW_URL`	`https://docuflow-api.neurolearninglabs.com`	Used for `file_path` / `url` inputs
`CONTEXTUALIZE_ENABLED`	`false`	Contextual retrieval (Anthropic pattern)
`CONTEXTUALIZE_MODEL`	`gpt-4.1-mini`	LLM used to prepend context summaries
`CONTEXTUALIZE_CONCURRENCY`	`16`	Parallel context generation requests
`CONTEXTUALIZE_MAX_CHARS`	`500000`	Max chars of document fed to LLM

⁠🚀 Quick Start — Docker Run

docker run -d \
  --name supabase-chunker \
  --restart unless-stopped \
  -p 8080:8080 \
  -e SUPABASE_URL="https://<ref>.supabase.co" \
  -e SUPABASE_SERVICE_ROLE_KEY="your-service-role-key" \
  -e CHUNKER_SERVICE_AUTH="your-32-byte-hex-secret" \
  -e EMBEDDING_API_KEY="your-embedding-api-key" \
  otakulabz/supabase-chunker:latest

Verify it's running:

curl http://localhost:8080/health
# → {"ok":true,"rate_limit":"100/60000ms","rate_used":0,"rag_table":"public.neuro_rag"}

⁠🐳 Docker Compose — With Traefik

services:
  researchlm-chunker:
    image: otakulabz/supabase-chunker:latest
    container_name: researchlm-chunker
    restart: unless-stopped
    env_file:
      - .env
    ports:
      - "8080:8080"
    environment:
      SUPABASE_URL: "${SUPABASE_URL}"
      SUPABASE_SERVICE_ROLE_KEY: "${SUPABASE_SERVICE_ROLE_KEY}"
      CHUNKER_SERVICE_AUTH: "${CHUNKER_SERVICE_AUTH}"
      # Optional overrides
      # EMBEDDING_MODEL: "text-embedding-3-small"
      # EMBED_BATCH_SIZE: "64"
      # EMBEDDING_BASE_URL: "https://api.voidai.app/v1"
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:8080/health"]
      interval: 30s
      timeout: 5s
      retries: 3
    networks:
      - dokploy-network
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.researchlm-chunker.rule=Host(`chunker.your-domain.com`)"
      - "traefik.http.services.researchlm-chunker.loadbalancer.server.port=8080"
      - "traefik.http.routers.researchlm-chunker.entrypoints=websecure"
      - "traefik.http.routers.researchlm-chunker.tls=true"
      - "traefik.http.routers.researchlm-chunker.tls.certresolver=cloudflare"
      - "traefik.http.routers.researchlm-chunker.service=researchlm-chunker"
      - "traefik.docker.network=nat"

networks:
  nat:
    external: true

⁠🐳 Docker Compose — Minimal (no reverse proxy)

services:
  supabase-chunker:
    image: otakulabz/supabase-chunker:latest
    container_name: supabase-chunker
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      SUPABASE_URL: "https://<ref>.supabase.co"
      SUPABASE_SERVICE_ROLE_KEY: "your-service-role-key"
      CHUNKER_SERVICE_AUTH: "your-32-byte-hex-secret"
      EMBEDDING_API_KEY: "your-embedding-api-key"
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:8080/health"]
      interval: 30s
      timeout: 5s
      retries: 3

⁠🔗 Wire Into Supabase Edge Functions

Set these secrets in your Supabase project so process-document-callback and reindex-source pick them up automatically:

supabase secrets set CHUNKER_SERVICE_URL=https://chunker.your-domain.com
supabase secrets set CHUNKER_SERVICE_AUTH=<same value as your service>

⁠✅ Verify End-to-End

# 1. Pull from DB
curl -X POST https://chunker.your-domain.com/chunk-source \
  -H "Authorization: Bearer $CHUNKER_SERVICE_AUTH" \
  -H "Content-Type: application/json" \
  -d '{"source_id":"f0945db8-cf9b-4798-bbca-9ee5ed7f8fc6"}'

# 2. Raw markdown with agent tagging
curl -X POST https://chunker.your-domain.com/chunk-source \
  -H "Authorization: Bearer $CHUNKER_SERVICE_AUTH" \
  -H "Content-Type: application/json" \
  -d '{
    "markdown": "# Own the Day\n\nAubrey Marcus...",
    "title": "Own the Day — Aubrey Marcus",
    "tagged_by": ["ASAGI"],
    "topics": ["health","productivity"],
    "agent_notebook": "ASAGI"
  }'

# 3. Local file
curl -X POST https://chunker.your-domain.com/chunk-source \
  -H "Authorization: Bearer $CHUNKER_SERVICE_AUTH" \
  -H "Content-Type: application/json" \
  -d '{"file_path":"/data/uploads/owntheday.pdf","tagged_by":["ASAGI"]}'

# 4. Remote URL
curl -X POST https://chunker.your-domain.com/chunk-source \
  -H "Authorization: Bearer $CHUNKER_SERVICE_AUTH" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/doc.pdf","agent_notebook":"Rias Gremory"}'

⁠📊 Rate Limiting

The service enforces a sliding-window rate limit on embedding API calls. With defaults (EMBED_RATE_LIMIT=100, EMBED_RATE_WINDOW_MS=60000), at most 100 embedding requests are dispatched per 60-second window regardless of concurrency. If the window fills, callers sleep until a slot opens — no dropped requests, no 429s propagated upstream.

Current usage is visible on GET /health:

{ "ok": true, "rate_limit": "100/60000ms", "rate_used": 12, "rag_table": "public.neuro_rag" }

Tune for your provider's quota:

# Stricter provider
EMBED_RATE_LIMIT=50

# Generous quota
EMBED_RATE_LIMIT=200

⁠💡 Why This Exists

Supabase Edge Functions hit WORKER_RESOURCE_LIMIT (200ms CPU cap on Free tier) when processing large documents
An earlier n8n implementation had a per-item pairing bug that silently dropped 24 of every 25 chunks
This service runs the same heading-aware chunking algorithm on external infrastructure with neither constraint

⁠🏷️ Image Details

Property	Value
Base image	`denoland/deno:alpine-2.1.4`
Image size	~54.9 MB
Exposed port	`8080`
Runtime user	`deno` (non-root)
Permissions	`--allow-net --allow-env --allow-read`

⁠📎 Source

gitlab.neurolearninglabs.com/code-labz/supabase-chunker⁠

Tag summary

Recent tags

Content type

Image

Digest

sha256:6c0fef4eb…

Size

54.9 MB

Last updated

5 months ago

docker pull otakulabz/supabase-chunker