hackerdogs/hdsearch repository overview

⁠✨ Features

• 🔎 Meta-search across 20+ engines — SearXNG⁠, OpenSERP⁠ (Google/Yandex/Baidu/Bing), DuckDuckGo⁠, Wikipedia⁠, and more. Prioritized fallback (free/self-hosted first) or aggregate mode with cross-engine dedup and facets. Per-user enable/disable + per-modality ranking of engines in the UI.
• 🗂️ Search modalities — web, news, images, videos, maps / places, scholar, code, social, web archive (Wayback⁠ + Common Crawl⁠), darkweb (Ahmia⁠ over Tor⁠), semantic (vector KNN grounded with live web), and agentic AI Search.
• 🕷️ Crawl — fetch any URL to clean markdown / links / text via self-hosted Crawl4AI⁠ + a Browserless⁠ headless-Chrome fallback for JS pages (optional Firecrawl⁠).
• 🧠 Vector search + RAG — index documents (TTL’d) and run semantic KNN via Redis RediSearch⁠ (HNSW), with a brute-force fallback. Upload your own files → parse → embed → retrieve, with citations.
• 🤖 AI Search (agentic) — a chat that plans and calls tools (search, maps, crawl, archive, chart, weather, render UI) and streams the answer. Threads, folders, and file attachments persist (Redis + S3); temporary chat skips server history. Works with local Ollama⁠ (no API key, $0) or any commercial model you add. Also exposes an OpenAI-compatible /v1/openai/chat/completions endpoint.
• ⏱️ Cache & history TTLs — Redis result-cache TTL (user preference, admin default + hard max) and history / AI-thread TTL (1–30 days). Temporary search opts a query out of server history.
• 📈 Trends — /trends headlines panel (platform defaults; optional hd-feeds⁠ integration).
• 🔌 MCP⁠ server — expose hd_search, hd_crawl, hd_vector_search, hd_vector_index, hd_list_engines to Claude / any MCP client (Streamable HTTP on :8792, or stdio).
• 🔐 Bring-your-own keys, encrypted — commercial engine/LLM keys are entered in the UI and stored AES-256-GCM encrypted in your DB. Nothing leaves your infra. Admins can also set system-wide default keys.
• 📦 Truly self-contained — one docker compose up bundles Postgres⁠/TimescaleDB⁠, Redis⁠, SeaweedFS⁠, embeddings, and every provider. No accounts, no SaaS, no per-request bills for the aggregator.

⁠🚀 Quickstart (2 minutes)

You need Docker (with Compose). Nothing else.

Pick a deployment topology — all on the hdsearchnet network. The helper scripts (./start_docker.sh, ./stop_docker.sh) are thin wrappers; you can always run docker compose directly.

Full — everything in one command (recommended to start):

git clone https://github.com/hackerdogs-ai/hdsearch.git && cd hdsearch
docker compose -f docker-compose-full.yml up -d
open http://localhost:3000                          # first run → setup wizard, then admin account
# stop:  docker compose -f docker-compose-full.yml down
# optional wrappers: ./start_docker.sh · ./stop_docker.sh

Local development (API / web on the host, against Docker infra or your own services):

./start_api.sh          # API on :8791 (tsx watch)
./start_web.sh 3005     # Web UI (pick a free port)
# stop: ./stop_local.sh

Split — run infra and core separately (scale them independently):

docker compose -f docker-compose-infra.yml up -d    # datastores + providers
docker compose -f docker-compose-core.yml up -d     # api + web

Core only — point HD-Search at services you already run (Postgres, Redis, S3, …):

docker compose -f docker-compose-core.yml up -d     # then open the wizard and enter your endpoints

No env vars required — the setup wizard (below) collects and verifies your endpoints in the browser. You can still pre-seed them with HDSEARCH_DATABASE_URL / HDSEARCH_REDIS_URL / HDSEARCH_S3_ENDPOINT if you prefer a headless bootstrap.

⁠First-run setup wizard

On first visit every page redirects to /setup — an OS-installer-style wizard that connects HD-Search to its infrastructure:

1. Welcome → 2. Datastores (Postgres, Redis, S3, embeddings) → 3. Providers (SearXNG, OpenSERP, Crawl4AI, Browserless, Tor) → 4. Review & finish.

• Every field defaults to the bundled container name on hdsearchnet (hds-db, hds-redis, …), so the Full/Split stacks are all green out of the box — just click through.
• Each service has a Test button that runs a live reachability probe. Required services (Database, Redis) must be reachable — the Next button stays greyed until they are.
• Running your own datastores? Replace the defaults, click Test, continue.
• Finishing writes the endpoints to a config volume and marks setup complete; the gate lifts and you proceed to create the admin account (or set HDSEARCH_ADMIN_EMAIL / HDSEARCH_ADMIN_PASSWORD for a headless bootstrap).
• Change endpoints later at System Admin → Infrastructure (admin-only). Infrastructure changes take effect after docker restart hds-api.

That's the whole setup. There are no secrets to configure — the app auto-generates its crypto secrets on first boot.

⁠Accounts

• First run → create the admin. With no accounts yet, the sign-in page shows "Create your admin account"; the account you create becomes the administrator.

• Self-service sign-up is open by default — the sign-in page has a Create an account link (email, password, confirm) that creates a normal user.

• Lock it down anytime at System Admin → User registration: flip to invite-only and the sign-up form disappears (admins create accounts). Or set HDSEARCH_OPEN_SIGNUP=false at deploy time.

• Optional SMTP (System Admin → Email) unlocks email verification, forgot / reset password, and magic-link sign-in. Without SMTP, local password auth still works.

• After sign-in, a one-time disclaimer gate records consent before the app unlocks.

• Add search/LLM provider keys later, in the UI — Account → Provider Keys (per-user) or System Admin (system-wide). Rank engines under Dashboard → Services. The free/self-hosted engines and local Ollama work with no keys.

• The API also serves the MCP server (Streamable HTTP) on :8792, in the same container — no separate process to run.

• Build from source instead of pulling images: ./publish_to_docker.sh --build-only --native <ns> then run any compose above (they use <ns>/hdsearch:*).

⁠Optional: self-hosted Firecrawl

Firecrawl⁠ (AGPL-3.0) is available as an extra crawl provider. It is opt-in, not part of the default stack — it brings its own Postgres, Redis, RabbitMQ and Playwright service and wants ~8 GB RAM, while the bundled crawl4ai + browserless pair already covers most crawling.

docker compose -f docker-compose-firecrawl.yml up -d          # start it on hdsearchnet
HDSEARCH_FIRECRAWL_URL=http://hds-firecrawl:3002 \
  docker compose -f docker-compose-full.yml up -d hds-api      # point HD-Search at it

A self-hosted instance runs with auth disabled, so no API key is needed — HD-Search detects the local endpoint and stops asking for the firecrawl credential. You can also set the endpoint in the setup wizard or System Admin → Infrastructure.

Worth knowing: Firecrawl's anti-bot engine is closed-source and cloud-only, so a self-hosted instance has no anti-bot capability and uses your own proxies.

⁠Try it

# issue an API key: Account → API Keys in the UI, or:
docker compose -f docker-compose-full.yml exec hds-api node dist/scripts/hds-keys.js issue --user me --name laptop

KEY=sk-hds-...
curl http://localhost:8791/v1/search -H "authorization: Bearer $KEY" \
  -H 'content-type: application/json' \
  -d '{"q":"open source search","mode":"aggregate","facets":true}'

⁠🧩 Architecture

                          ┌─────────────────────────── hdsearchnet (private) ───────────────────────────┐
  Browser ─▶ hds-web ─▶ hds-api ─▶ engine ─▶ providers  (searxng, openserp, crawl4ai, browserless, tor) │
             (Next.js)   (Hono)  │  ├─ Redis Stack  ── cache · history · threads · RediSearch vectors    │
                                 │  ├─ Postgres/TimescaleDB ── users · encrypted keys · usage            │
                                 │  ├─ SeaweedFS (S3) ── crawl archive · AI files · thread archive       │
                                 │  ├─ embeddings (MiniLM-384) ── vectors for search + RAG               │
                                 │  └─ Ollama (host) ── local LLMs for AI Search (no key, $0)            │
  MCP client ─▶ :8792 /mcp ──────┘                                                                       │
                          └────────────────────────────────────────────────────────────────────────────┘

Every backend is bundled and internal-only. The API degrades gracefully if an optional dependency is down (search still works without Postgres; vector falls back to brute-force without RediSearch, etc.).

⁠📚 API

Base URL http://localhost:8791. Auth: Authorization: Bearer sk-hds-…. OpenAPI at GET /openapi.json; interactive Swagger UI at /api (public) and /dashboard/api-reference (signed-in). In-app docs: /docs.

{ "query":"open source search","total":16,"cached":false,"tookMs":1254,
  "engines":[{"engine":"searxng","ok":true,"count":16}],
  "results":[{"title":"…","url":"https://…","snippet":"…","engine":"searxng"}],
  "facets":{"engine":[…],"site":[…]} }

⁠🤖 AI Search (agentic)

A chat that plans, calls tools, and streams a cited answer. Tools: hd_search, hd_maps, hd_plot_map, hd_crawl, hd_archive, hd_chart, hd_weather, and hd_render (rich UI components).

• Local & free by default: point it at a host-run Ollama⁠ — models are auto-discovered from whatever you've pulled, no API key, $0. The compose already wires host.docker.internal:11434.
• Or any commercial model — add the key in the UI and pick the model from the dropdown.
• Threads & folders: conversations persist server-side (Redis hot tier + S3 archive) with the same history TTL as search; organize them into folders; temporary chat keeps a session local-only.
• RAG: upload files to a chat → they're parsed, embedded, indexed, and retrieved to ground the answer with citations (delete cascades with the thread).
• Tokens, not credits: each answer reports the tokens it actually consumed. No metering, no quotas, no per-request billing by hdsearch.

⁠LLM providers

Ten providers ship in the registry. Only Ollama is required for AI Search to work — everything else is optional and needs your own key, entered in the UI and stored AES-256-GCM encrypted.

Add your own. System Admin → LLM providers takes any OpenAI-compatible endpoint — vLLM⁠, LM Studio⁠, llama.cpp server⁠, Together⁠, Fireworks⁠, DeepSeek⁠, Perplexity⁠ — by base URL alone, no code change. Register its models under AI Search models on the same page. Both are stored in Postgres and take effect without a restart.

Model metadata (context window, max output, tool/vision/thinking support) is admin-editable, so you can add a model the day it ships rather than waiting on a release.

⁠🔌 MCP server

The stack runs a Streamable HTTP MCP server at http://localhost:8792/mcp. Connect any MCP client (Claude, Cursor, …) — each caller authenticates with their own key via the Authorization header (so their scopes and rate limits apply):

{ "mcpServers": { "hd-search": {
  "type": "streamable-http",
  "url": "http://localhost:8792/mcp",
  "headers": { "Authorization": "Bearer sk-hds-…" }
}}}

Prefer a local process (stdio)? That works too:

{ "mcpServers": { "hd-search": {
  "command": "node", "args": ["dist/mcp/server.js"],
  "env": { "HDSEARCH_API_URL": "http://127.0.0.1:8791", "HDSEARCH_API_KEY": "sk-hds-…" }
}}}

Tools: hd_search, hd_crawl, hd_vector_search, hd_vector_index, hd_list_engines.

⁠⚙️ Configuration

.env.selfhost / .env is optional and contains no secrets — only ports, URLs, and flags. Copy .env.selfhost.example to change a default.

SMTP (SMTP_HOST / SMTP_USER / SMTP_PASSWORD / SMTP_FROM, or System Admin → Email) is optional — it enables verification, password reset, and magic-link sign-in.

Provider keys and all secrets are handled in the UI or auto-generated — never in env. See docs/CONFIGURATION_DEPLOYMENT.md⁠.

⁠🔐 Security & privacy

• Your data stays on your infra. No telemetry, no external SaaS, no per-request calls home.
• No secrets in files. Encryption / session / internal-BFF secrets are auto-generated into a Docker volume; back it up for disaster recovery.
• Provider keys are encrypted at rest (AES-256-GCM) in your Postgres — plaintext keys never touch disk.
• Local auth — email + password (scrypt-hashed), first-run admin, DB-driven roles (admin/user). Sessions are an encrypted, httpOnly cookie. With SMTP: email verification, password reset, magic-link sign-in.
• Privacy controls — temporary search/chat, configurable history TTL, and a one-time disclaimer acceptance gate.
• Rate limiting per identity (default 120/min) with X-RateLimit-* headers; CORS is configurable.

⁠⚠️ Limitations & caveats

• Free search engines vary. Google via OpenSERP frequently hits CAPTCHAs from datacenter IPs — results lean on Yandex/Baidu/SearXNG unless you add residential proxies or a 2captcha key (docs/OPENSERP.md⁠). This is inherent to scraping, not a bug.
• AI answer quality tracks the model. A small local Ollama model gives weaker open-ended answers than a frontier model; the tool-calling pipeline is the same. Add a commercial key for best results.
• Darkweb/maps/scholar coverage depends on upstream availability and may return empty cleanly.
• First up is heavy — it pulls a few GB (embeddings, browsers, providers). The embeddings image is amd64 (emulated on Apple Silicon).

Full README, development guide and roadmap: https://github.com/hackerdogs-ai/hdsearch⁠