Sign inSign up

techidiotsllc/pmtiles-swarm

By techidiotsllc

•Updated 18 days ago

BitTorrent distribution and tile serving for PMTiles archives

Image
0

145

techidiotsllc/pmtiles-swarm repository overview

⁠Docker

docker run -d --name pmtiles-swarm \
  --network host \
  -v /srv/maps:/data \
  techidiotsllc/pmtiles-swarm:latest

Images are techidiotsllc/pmtiles-swarm, built for amd64 and arm64, tagged v<version> with latest following the newest stable release. A prerelease never becomes latest.

⁠One directory

/data is the only mount that matters. On first start, if there is no configuration there, the container writes one:

{
  "dataDir": "./state",
  "savePath": "./archives",
  "watch": [{ "path": "./generated", "categories": ["basemaps"] }],
  "libtorrent": { "resumeDir": "./state/resume" }
}

Every path is relative, and a pmtiles-swarm configuration resolves paths against the configuration file rather than against the working directory. So they all land inside whatever you mounted, the file says nothing about containers, and the same directory works unchanged if you later run the node on a host instead.

/srv/maps/
  swarm.config.json     yours to edit; never overwritten once it exists
  state/                catalog, stored .torrent files, resume data
  state/resume/
  archives/             what downloads land in
  generated/            watched: archives dropped here are published

Edit the configuration after the first start. It ships with placeholders for the API key and the console password, which are not credentials.

⁠The one way this bites

An absolute path in a mounted configuration refers to a path inside the container. "path": "/mnt/store/generated" does not reach your host — it names a directory that does not exist in the image, and a watched folder that does not exist looks exactly like a folder nothing ever arrives in. Keep paths relative, or mount the host directory at that exact absolute path.

⁠Networking is not the usual Docker question

This is a BitTorrent peer, not only an HTTP server, and bridge networking breaks it in three ways at once:

  • Incoming connections. Peers cannot reach a port that is NAT-ed, so the node only ever makes outgoing connections and reads as though it is in an empty swarm.
  • The DHT. It tells other peers where to find you, and behind bridge NAT it advertises an address nothing can reach.
  • UPnP and NAT-PMP. They ask the router for a forward. From inside a bridge network, they ask the wrong device.

--network host avoids all three, at the cost of the isolation containers are usually for. That is the trade, stated plainly, and it is why the compose file uses it.

Bridge networking does work, and seeding to peers you connect to is unaffected — expect fewer peers, and forward the port on the router yourself:

-p 8090:8090 -p 8091:8091 \
  -p 6881:6881/tcp -p 6881:6881/udp \
  -p 6882:6882/tcp -p 6882:6882/udp     # only with a WebTorrent secondary

Both protocols. uTP and the DHT are UDP; publishing only TCP is a common mistake that leaves half of BitTorrent unreachable while looking correct.

⁠Ports

8090Tiles, feeds, TileJSON, the .torrent endpoints — the public surface
8091Console and API. Do not expose this one to the internet
6881libtorrent, TCP and UDP
6882WebTorrent, TCP and UDP — only when secondaryEngines is set
6883UDP, the BEP 46 publisher's DHT — only when mutable.dhtPort says so, and optional even then

One peer port per engine. Each engine is its own listener and two cannot share a port, so a node running libtorrent with WebTorrent alongside it needs both. The default configuration this image writes runs libtorrent only, and 6882 is idle until you add:

{
  "secondaryEngines": ["webtorrent"],
  "webtorrent": { "clientOptions": { "torrentPort": 6882 } }
}

WebRTC needs no port. Browser peers are reached through a wss:// tracker and ICE, which is outbound only — there is nothing to forward for them, and nothing that a reverse proxy carries.

The publisher's DHT does not need one either. A container that publishes BEP 46 records works with no inbound port at all: a put is outbound, and the replies come back on the same socket the way any UDP client's do. Pin and publish 6883 only to make this a reachable DHT node, which earns a better routing table and contributes back:

{
  "mutable": {
    "publish": true,
    "keyPath": "/data/publisher.pem",
    "dhtPort": 6883
  }
}

Note that this is a third UDP socket. Each engine runs a DHT of its own on its peer port, so 6883 has to be clear of both — two sockets cannot hold one port, and the container will fail to start if they collide.

The key belongs on a mount, not in the image. /data/publisher.pem, created with

docker run --rm techidiotsllc/pmtiles-swarm publisher-key > /path/to/data/publisher.pem
chmod 400 /path/to/data/publisher.pem

and readable by the uid the container runs as. Baking it into an image would ship your signing identity to anyone who pulls it. Only the node that builds needs it; see docs/serving-tiles.md⁠.

⁠Permissions

The image runs as uid 10001, so the mounted directory has to be writable by it:

sudo chown -R 10001:10001 /srv/maps

Or run as whoever owns it, with --user "$(id -u):$(id -g)". The container cannot guess, and a node that cannot write its data directory fails at the first thing it tries to save.

⁠What this image is for

A subscriber: a node that follows a feed, mirrors or caches archives, and serves tiles. That is the deployment worth containerising, and the one you would want several of behind a load balancer.

A worker: a machine lending its CPU to another node's exports. Nothing to configure and nothing to expose, which makes it the easiest of the three to run anywhere — see below⁠.

A publisher that generates its own archives is a poorer fit. An onComplete hook runs its command inside the container, so a build toolchain — planetiler, a JVM, its heap — would have to be in the image alongside the node. Running that one as a service on the host is the better arrangement; see running as a service⁠.

⁠As a worker

The same image lends a machine to another node's exports. A worker is not a node: it writes no configuration, opens no ports and holds no library, so none of the above applies to it — only an address and a token.

docker run -d --name pmtiles-swarm-worker --restart unless-stopped \
  -e PMTILES_SWARM_COORDINATOR=http://maps.lan:8090 \
  -e PMTILES_SWARM_WORKER_TOKEN=<worker token> \
  -e PMTILES_SWARM_WORKER_NAME=rack-2 \
  -v pmtiles-swarm-worker:/data \
  techidiotsllc/pmtiles-swarm:latest worker

Settings → Workers on the coordinator mints the token and shows this command with it filled in. The volume is optional and worth having: shards in progress are kept under /data/worker, so a container restarted mid-shard picks up where it was if the same shard comes back to it, and it keeps the identity the coordinator lists it by. Bridge networking is fine — a worker only makes outgoing connections. See workers.md⁠.

⁠Updating

docker compose pull && docker compose up -d

Nothing under /data is touched, and a clean stop writes resume data, so the node comes back without re-hashing its archives.

Tag summary

Content type

Image

Digest

sha256:477ef5366…

Size

174.9 MB

Last updated

18 days ago

docker pull techidiotsllc/pmtiles-swarm