parsehub/iute repository overview

⁠iute

⁠Description

iute is an all-in-one distributed docker registry with next to no coding involved.

⁠Container contents

iute comes bundled with:

1. etcd⁠, a distributed key-value store for locking and discovery
2. syncthing⁠, open-source bittorrent file syncing software
3. nginx⁠, a reverse-proxy with auth capabilities
4. docker distribution⁠, the "docker registry" in registry:2

⁠Motivation

At ParseHub⁠ we wanted a way to reduce our docker push to docker pull round trip time, which was bottlenecked on a single build server which pushed to a single registry which was pulled from O(n) servers at once.

Instead, we came up with a system where a cluster of iute registries were running throughout our production and staging machines.

This uses the powerful BitTorrent⁠ file distribution system to make the docker push speed equal to pushing locally, and the docker pull speed equivalent to pulling locally.

We use etcd to implement a distributed Readers-writer lock⁠ to ensure that no registries are being pulled from at the same time that a registry is being pushed to.

This is a reasonable compromise in our use case, which is push THEN pull exclusively.

⁠Usage

1. Clone this repository with git clone https://github.com/parsehub/iute.git
2. Build the docker image with docker build -t 'iute:latest' .
3. Distribute iute to the first host, then docker run --rm -i iute:latest --create localhost:10000-10006 > bundle.ut, load the bundle with docker run --name iute -p 10001-10006:1-6,8000:8000 -i iute:latest:10001-10006 --registry-port 8000 localhost < bundle.ut, and start iute with docker start iute
4. Copy the bundle over to all servers which are to be part of the cluster, then repeat the former two commands (with potentially modified ports) to load the bundle and start iute on all of the hosts.

⁠Local usage example (two registries on same host)

docker rm -f iute
docker build -t iute:1.0 . && \
docker run --rm -i iute:1.0 --create localhost:10000-10005 > bundle.ut && \
docker run -p 10000-10005:10000-10005 -p 8000:8000 --network host --name iute -i iute:1.0 localhost:10000-10005 -r 8000 < bundle.ut && \
docker start iute

docker rm -f iute2
docker run -p 10006-10011:10006-10011 -p 8000:8000 --network host --name iute2 -i iute:1.0 localhost:10006-10011 < bundle.ut && \
docker start iute2