fredfeng033/greenfinger-api repository overview

⁠Greenfinger

One url in. A searchable archive out.

A distributed web crawler for the JVM. One pass writes plain files, a full text index and a vector collection. Every node runs the same jar, there is no coordinator to deploy, and the first crawl needs no database, no search server and no API key.

This image is one crawler node. For the console in front of it, see fredfeng033/greenfinger-ui⁠.

• Source: https://github.com/paganini2008/greenfinger⁠
• Documentation: https://paganini2008.github.io/greenfinger/⁠
• Licence: Apache 2.0

⁠Quick start

One node, nothing else running:

docker run -d --init --name greenfinger \
  -p 50080:50080 \
  -v greenfinger-data:/app/data \
  fredfeng033/greenfinger-api:latest

Then open http://localhost:50080. The jar serves the console itself, so a single container is a complete installation. Sign in with admin and admin123.

Give the container 2GB. Both the ONNX embedding models and the headless chromium live off heap, so a 1g node is killed the first time it meets either.

For three nodes with one address in front of them, see Running a cluster below.

⁠What it does

⁠Tags

Both linux/amd64 and linux/arm64. The same images are on GHCR as ghcr.io/paganini2008/greenfinger-api.

⁠Running a cluster

Nodes agreeing on the cluster name and port find each other and share one crawl. Every node is told the whole peer list including itself, so they can start in any order.

Save this as docker-compose.yml:

name: greenfinger

# Three crawler nodes and one console in front of them. Static addresses because the cluster
# setting is a list of ip addresses, so a node has to advertise something its peers can dial.
x-api: &api
  image: fredfeng033/greenfinger-api:latest
  restart: unless-stopped
  init: true      # reaps the headless browser's helper processes
  mem_limit: 2g   # the embedding models and chromium are both off heap
  environment: &api-env
    GF_CLUSTER_NAME: greenfinger
    GF_CLUSTER_PORT: '22020'
    GF_CLUSTER_HOSTS: 172.28.0.10,172.28.0.11,172.28.0.12
    # Tokens are signed, not looked up, so every node needs the same key or it rejects the
    # others' sign-ins and the console logs you out on two requests in three.
    GF_TOKEN_SECRET: "${GF_TOKEN_SECRET:?missing. Run - echo GF_TOKEN_SECRET=$$(openssl rand -base64 32) > .env}"
  networks:
    cluster:

services:
  api-1:
    <<: *api
    environment:
      <<: *api-env
      GF_CLUSTER_ADVERTISE_HOST: 172.28.0.10
    networks:
      cluster:
        ipv4_address: 172.28.0.10
    ports:
      - "50080:50080"   # one node published, for curl. The console does not need it
    volumes:
      - api-1-data:/app/data

  api-2:
    <<: *api
    environment:
      <<: *api-env
      GF_CLUSTER_ADVERTISE_HOST: 172.28.0.11
    networks:
      cluster:
        ipv4_address: 172.28.0.11
    volumes:
      - api-2-data:/app/data

  api-3:
    <<: *api
    environment:
      <<: *api-env
      GF_CLUSTER_ADVERTISE_HOST: 172.28.0.12
    networks:
      cluster:
        ipv4_address: 172.28.0.12
    volumes:
      - api-3-data:/app/data

  ui:
    image: fredfeng033/greenfinger-ui:latest
    restart: unless-stopped
    environment:
      GF_UPSTREAMS: 172.28.0.10:50080,172.28.0.11:50080,172.28.0.12:50080
    ports:
      - "9700:80"
    networks:
      cluster:
    depends_on: [api-1, api-2, api-3]

networks:
  cluster:
    ipam:
      config:
        - subnet: 172.28.0.0/16

volumes:
  api-1-data:
  api-2-data:
  api-3-data:

Then:

echo "GF_TOKEN_SECRET=$(openssl rand -base64 32)" > .env
docker compose up -d
open http://localhost:9700

Or take the annotated version, which has commented blocks for PostgreSQL, MySQL, Elasticsearch, Qdrant and MinIO:

curl -O https://raw.githubusercontent.com/paganini2008/greenfinger/main/backend/greenfinger-docker/docker-compose.yml

Four things in there are easy to get wrong:

Give Docker 6GB for this. A warm node holds about 1.7GB. Short of it the host kernel kills whichever jvm it reaches first, and because that is the host rather than the container's own limit, docker reports neither an OOM nor a non zero exit: all you see is containers restarting with nothing in the log. One node and the console fit in 2GB.

⁠Configuration

Nothing below is required. H2 and an embedded Lucene index are the defaults.

⁠Metadata store

⁠Full text and vectors

⁠Files

⁠Embeddings

⁠Cluster

⁠Crawl defaults

Every one of these is also a per catalog setting, and the catalog wins.

⁠Server

⁠Volumes and ports

⁠Memory

GF_MEMORY or --memory caps heap and off heap together, and both the ONNX embedding models and a headless chromium live off heap.

The 1.5g row is the one worth knowing about, because a node looks fine at 1.5g until the image model arrives: the text model loads, the node reports healthy, joins the cluster and takes work, and the kill comes a minute later on a thread called greenfinger-emb.

Give a node 2g, and give it more if it is doing both at once.

That is the container's own limit. The host needs the sum of them: a warm node settles around 1.7GB, so three nodes and the console want 6GB of Docker memory. When the host runs short it is the host kernel that kills a jvm, not the container's limit, and docker then reports neither an OOM nor a non zero exit. A container that restarts every minute or two with nothing in its log is this, every time.

⁠Own your own accounts

Two roles, in users.xml. ADMIN can crawl, edit, replay and delete. SUPPORT can read every page and is offered no button that writes.

docker run -d --init \
  -v ./users.xml:/app/config/api/users.xml:ro \
  -v greenfinger-data:/app/data \
  -p 50080:50080 \
  fredfeng033/greenfinger-api:latest

<users>
  <user name="admin"  password="..." roles="ADMIN"/>
  <user name="tester" password="..." roles="SUPPORT"/>
</users>

Apache License 2.0. Issues and pull requests at https://github.com/paganini2008/greenfinger⁠.