Public | Automated Build

Last pushed: 2 years ago
Short Description
Reverse caching proxy for distro mirrors
Full Description

Lazy Distro Mirrors

I have a pretty slow internet connection. I don't run a computer lab.
I don't have users. But, I do have multiple machines running various
distros of Linux. If I update them, they tend to download a lot of
redundant data, which is taxing on my connection. I could run a local
mirror of the Arch and Debian repositories I use, but that seems like
overkill. Really what I want is to just be able to cache files that I
download once so that I don't have to re-download them a second time.
This is the job for a caching proxy server, like squid. Except, HTTP
proxies are a pain to setup, especially with package managers. I want
something that just works.

Lazy Distro Mirrors is a docker container that hosts a reverse squid
proxy. This is effectively a local mirror of a remote website (like
http://mirrors.kernel.org), with transparent caching. This is not a
full mirror, but it looks and behaves like one. When you download a
file for the first time, it goes to the remote mirror to fetch it.
Subsequent retrievals will fetch it from the local cache. It will do
this until you fill up the cache (20GB by default) and then it will
start deleting the least recently used file.

The benefit of a reverse proxy is that you don't have to configure an
HTTP proxy, instead you edit the package manager configuration to use
a local URL for the mirror to use. The hard configuration is done once,
on one computer, and then all the rest of the computers on your LAN
get to use it by just choosing the correct mirror URL.

Thank you to Matt Wagner and his blog post
. I copied most of his squid configuration for this. I also copied
the name he used, as I couldn't think of anything better. Being lazy
seemed appropriate.

Usage

You need docker, install it however you like. Here's the official docs
for that
.

You'll need two directories, one to place the config file, and one for
the squid cache directory. For example:

mkdir -p /opt/docker/lazy-distro-mirrors/config
mkdir -p /opt/docker/lazy-distro-mirrors/cache

Download and start the docker container:

docker run \
--name lazy-distro-mirrors \
-d --restart=always \ 
--publish 8080:8080 \
--volume /opt/docker/lazy-distro-mirrors/config:/docker_configurator/ \
--volume /opt/docker/lazy-distro-mirrors/cache:/var/spool/squid3 \
enigmacurry/lazy-distro-mirrors

Replace with the directory names you chose. Replace the first 8080
with the port you want the proxy to run on.

By now the proxy should be running on the port you chose.

On first running the container, it will have written it's config file
to the config directory you specified. Open up config.yaml in that
location.

You'll see some settings you may want to edit:

mirrors:
 kernel-mirror: http://mirrors.kernel.org
 xmission-mirror: http://mirrors.xmission.com
 advancedhosters-mirror: http://mirrors.advancedhosters.com

These are the three default mirrors I use, but you can add your own
here. Each mirror has a name and a URL. Just edit the file, save it,
and restart the container:

docker restart lazy-distro-mirrors

Finally, Squid relies upon name-based resolution to pick the mirror
you're using. For instance, to use the kernel.org mirror, you'll need
to go to http://kernel-mirror:8080/ - this is the local URL for that
mirror. If you have a DNS server running on your router, add that IP
of your docker container and assign it the name kernel-mirror. If
you don't run your own DNS server, you can also just add it to your
/etc/hosts file:

127.0.0.1      kernel-mirror xmission-mirror advancedhosters-mirror

Do the same for all mirrors you define.

Now you can switch between mirrors by going to the different local
URLs:

http://kernel-mirror:8080/
http://xmission-mirror:8080/
http://advancedhosters-mirror:8080/

Package manager configuration

In order to get your distro to use this for package updates, you'll
need to select the local URL as your primary mirror location. Consult
your distribution's documentation for how best to do that. Here's how
to do it briefly:

For Arch Linux, edit /etc/pacman.d/mirrorlist, put this at the top:

Server = http://kernel-mirror:8080/archlinux/$repo/os/$arch

If you want to force using this mirror, delete everything else in the
file.

For anything using apt-get, edit /etc/apt/sources.list and replace the
domain with your local one. For example:

deb http://us.archive.ubuntu.com/ubuntu/ trusty main restricted

Becomes:

deb http://kernel-mirror:8080/ubuntu/ trusty main restricted
Docker Pull Command
Owner
enigmacurry
Source Repository

Comments (2)
girishkg
a year ago

Please post the complete squid.conf content or attach it here. I am in need of it.

girishkg
a year ago

Hi,
This image is restarting repetedly.

(1) Start command (in single line) - All the volumes does exist:
docker run --name lazy-distro-mirrors -d --restart=always --publish 8080:8080 --volume /docker/ldm/config:/docker_configurator/ --volume /docker/ldm/cache:/var/spool/squid3 enigmacurry/lazy-distro-mirrors

(2) docker ps
root@fab-stg-repo-h1:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
04a6c881c0f6 enigmacurry/lazy-distro-mirrors "/start.sh" 8 minutes ago Restarting (2) 3 minutes ago 3128/tcp, 0.0.0.0:8080->8080/tcp lazy-distro-mirrors
root@fab-stg-repo-h1:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
04a6c881c0f6 enigmacurry/lazy-distro-mirrors "/start.sh" 8 minutes ago Restarting (2) Less than a second ago 3128/tcp, 0.0.0.0:8080->8080/tcp lazy-distro-mirrors
root@fab-stg-repo-h1:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
04a6c881c0f6 enigmacurry/lazy-distro-mirrors "/start.sh" 8 minutes ago Restarting (2) 8 seconds ago 3128/tcp, 0.0.0.0:8080->8080/tcp lazy-distro-mirrors
root@fab-stg-repo-h1:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
04a6c881c0f6 enigmacurry/lazy-distro-mirrors "/start.sh" 9 minutes ago Restarting (2) 13 seconds ago 3128/tcp, 0.0.0.0:8080->8080/tcp lazy-distro-mirrors

(3) docker info
root@fab-stg-repo-h1:~# docker info
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 1
Server Version: 1.11.1
Storage Driver: aufs
Root Dir: /docker/aufs
Backing Filesystem: extfs
Dirs: 25
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.2.0-27-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.86 GiB
Name: fab-stg-repo-h1
ID: J2TJ:O2XG:YYNT:COIK:KJZ2:J2U6:4WKS:RSVK:PN2P:5CMI:ABDX:P7PY
Docker Root Dir: /docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

(4) docker images
root@fab-stg-repo-h1:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
enigmacurry/lazy-distro-mirrors latest 606cb59c0612 3 months ago 238.9 MB