Public Repository

Last pushed: 17 days ago
Short Description
A minimal openresty image
Full Description

Building a Minimal Openresty Docker Image

We will build Openresty in a container, to avoid polluting the base installation.

docker run --rm \
       --name resty-builder \
       --hostname resty-builder \
       -v /var/run/docker.sock:/var/run/docker.sock \
       -it communitycloud/docker-builder

Within the builder container, we need a bunch of stuff to biuld. To get the latest
Postgres stuff, we'll need to add postgres sources first to apt, as described on the postgres wiki:

echo "deb http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
&& apt-get update && DEBIAN_FRONTEND=noninteractive \
&& apt-get install -y wget ca-certificates sudo \
&& wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - \
&& apt-get update \
&& apt-get install -y build-essential curl libreadline-dev libncurses5-dev \
                      libpcre3-dev libssl-dev lua5.2 luarocks perl tree \
                      libgeoip-dev libpq-dev

Now that we have all prerequisites, building Openresty is fairly straightforward. We'll use /opt as the root, with the hope that nothing ends up in the /usr folder.

wget http://openresty.org/download/ngx_openresty-1.7.7.2.tar.gz \
&& tar -xzf ngx_openresty-*.tar.gz \
&& rm -f ngx_openresty-*.tar.gz \
&& cd ngx_openresty-* \
&& ./configure \
    --with-http_ssl_module \
    --with-pcre \
    --with-pcre-jit \
    --with-ipv6 \
    --with-http_geoip_module \
    --with-http_gzip_static_module \
    --with-http_gunzip_module \
    --with-http_realip_module \
    --with-http_spdy_module \
    --with-http_ssl_module \
    --with-http_stub_status_module \
    --with-http_sub_module \
    --with-http_postgres_module \
    --with-http_secure_link_module \
    --with-sha1=/usr/include/openssl \
    --with-md5=/usr/include/openssl \
    --prefix=/opt/openresty \
    --sbin-path=/opt/openresty/sbin/nginx \
    --conf-path=/etc/openresty/nginx.conf \
    --error-log-path=/var/log/nginx/error.log \
    --http-log-path=/var/log/nginx/access.log \
    --pid-path=/var/run/nginx.pid \
    --lock-path=/var/run/nginx.lock \
    --http-client-body-temp-path=/var/cache/nginx/client_temp \
    --http-proxy-temp-path=/var/cache/nginx/proxy_temp \
    --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
    --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
    --http-scgi-temp-path=/var/cache/nginx/scgi_temp \
&& make \
&& make install \
&& cd .. \
&& rm -rf ngx_openresty-* \
&& ldconfig

Now that everything is nicely built and installed, this is what we have:

tree /opt
/opt
`-- openresty
    |-- bin
    |   `-- resty
    |-- luajit
    |   |-- bin
    |   |   `-- luajit-2.1.0-alpha
    |   |-- include
    |   |   `-- luajit-2.1
    |   |       |-- lauxlib.h
    |   |       |-- lua.h
    |   |       |-- lua.hpp
    |   |       |-- luaconf.h
    |   |       |-- luajit.h
    |   |       `-- lualib.h
    |   |-- lib
    |   |   |-- libluajit-5.1.a
    |   |   |-- libluajit-5.1.so -> libluajit-5.1.so.2.1.0
    |   |   |-- libluajit-5.1.so.2 -> libluajit-5.1.so.2.1.0
    |   |   |-- libluajit-5.1.so.2.1.0
    |   |   |-- lua
    |   |   |   `-- 5.1
    |   |   `-- pkgconfig
    |   |       `-- luajit.pc
    |   `-- share
    |       |-- lua
    |       |   `-- 5.1
    |       |-- luajit-2.1.0-alpha
    |       |   `-- jit
    |       |       |-- bc.lua
    |       |       |-- bcsave.lua
    |       |       |-- dis_arm.lua
    |       |       |-- dis_mips.lua
    |       |       |-- dis_mipsel.lua
    |       |       |-- dis_ppc.lua
    |       |       |-- dis_x64.lua
    |       |       |-- dis_x86.lua
    |       |       |-- dump.lua
    |       |       |-- p.lua
    |       |       |-- v.lua
    |       |       |-- vmdef.lua
    |       |       `-- zone.lua
    |       `-- man
    |           `-- man1
    |               `-- luajit.1
    |-- lualib
    |   |-- cjson.so
    |   |-- rds
    |   |   `-- parser.so
    |   |-- redis
    |   |   `-- parser.so
    |   `-- resty
    |       |-- aes.lua
    |       |-- core
    |       |   |-- base.lua
    |       |   |-- base64.lua
    |       |   |-- ctx.lua
    |       |   |-- exit.lua
    |       |   |-- hash.lua
    |       |   |-- misc.lua
    |       |   |-- regex.lua
    |       |   |-- request.lua
    |       |   |-- response.lua
    |       |   |-- shdict.lua
    |       |   |-- time.lua
    |       |   |-- uri.lua
    |       |   |-- var.lua
    |       |   `-- worker.lua
    |       |-- core.lua
    |       |-- dns
    |       |   `-- resolver.lua
    |       |-- lock.lua
    |       |-- lrucache
    |       |   `-- pureffi.lua
    |       |-- lrucache.lua
    |       |-- md5.lua
    |       |-- memcached.lua
    |       |-- mysql.lua
    |       |-- random.lua
    |       |-- redis.lua
    |       |-- sha.lua
    |       |-- sha1.lua
    |       |-- sha224.lua
    |       |-- sha256.lua
    |       |-- sha384.lua
    |       |-- sha512.lua
    |       |-- string.lua
    |       |-- upload.lua
    |       |-- upstream
    |       |   `-- healthcheck.lua
    |       `-- websocket
    |           |-- client.lua
    |           |-- protocol.lua
    |           `-- server.lua
    |-- nginx
    |   `-- html
    |       |-- 50x.html
    |       `-- index.html
    `-- sbin
        `-- nginx

29 directories, 70 files

There's some stuff here we won't need: resty command-line tool, luajit binary and headers, We will also need the passwd file with user nobody:

mkdir /opt/etc \
&& echo 'nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin' > /opt/etc/passwd \
&& echo 'nogroup:x:65534:' > /opt/etc/group \
&& echo 'nobody:*:16435:0:99999:7:::' > /opt/etc/shadow \
&& echo 'nogroup:*::' > /opt/etc/gshadow 

Resty is always built with all debug symbols included, to help development and production troubleshooting. We can strip those symbols and create a lighter version for production situations:

find /opt  -type f -not -name "*.lua" -and -not -name "*.html" -exec strip {} \; 2>/dev/null

Let's take what we need and tar it:

cd / \
&& tar cf resty.tar \
    /opt/openresty/luajit/share/luajit-2.1.0-alpha/jit/ \
    /opt/openresty/lualib/ \
    /opt/openresty/nginx/ \
    /opt/openresty/sbin/

In addition, we'll need all libraries that the nginx binary is linked to:

ldd /opt/openresty/sbin/nginx \
    | awk 'BEGIN{ORS=" "}$1~/^\//{print $1}$3~/^\//{print $3}'  \
    | sed 's/ /\n/g; s/_at\n/_at /g' \
    | tar rhf resty.tar -T -

This command figures out which libraries the binary is linked with and collectes them all into a tar file. The h flag in tar dereferences all symlinks and imports the target files instead.

Sadly, ldd does not always pick up all we need, as some dependencies get loaded dynamically depending on the container configuration. Typically, when the binary in the container needs to resolve a host from the hosts file, it will want to load libnss_files library, which does not come up in the above ldd output. If the container run fails, we have to run the container with strace and then redo the build proces. The below tar command contains the libraries we discovered by running the container with strace.

If we tar from the /opt directory, our passwd file will appear as /etc/passwd in the final image:

cd /opt \
&& tar rhf /resty.tar /lib/x86_64-linux-gnu/libnss_compat.so.2 \
                     /lib/x86_64-linux-gnu/libnss_files.so.2 \
                     /lib/x86_64-linux-gnu/libnsl.so.1 \
                     /lib/x86_64-linux-gnu/libnss_nis.so.2 \
                     /usr/lib/ssl/openssl.cnf /etc/ld.so.cache \
                     /etc/localtime /etc/nsswitch.conf \
                     etc/passwd etc/shadow etc/group etc/gshadow

Let's see the container layout now:

T=/tmp/resty && mkdir -p $T && tar xf /resty.tar -C $T && tree $T && du -sh $T && rm -rf $T

|-- etc
|   |-- group
|   |-- gshadow
|   |-- ld.so.cache
|   |-- localtime
|   |-- nsswitch.conf
|   |-- passwd
|   `-- shadow
|-- lib
|   `-- x86_64-linux-gnu
|       |-- libc.so.6
|       |-- libcom_err.so.2
|       |-- libcrypt.so.1
|       |-- libdl.so.2
|       |-- libgcc_s.so.1
|       |-- libkeyutils.so.1
|       |-- libm.so.6
|       |-- libnsl.so.1
|       |-- libnss_compat.so.2
|       |-- libnss_files.so.2
|       |-- libnss_nis.so.2
|       |-- libpcre.so.3
|       |-- libpthread.so.0
|       |-- libresolv.so.2
|       `-- libz.so.1
|-- lib64
|   `-- ld-linux-x86-64.so.2
|-- opt
|   `-- openresty
|       |-- luajit
|       |   |-- lib
|       |   |   `-- libluajit-5.1.so.2
|       |   `-- share
|       |       `-- luajit-2.1.0-alpha
|       |           `-- jit
|       |               |-- bc.lua
|       |               |-- bcsave.lua
|       |               |-- dis_arm.lua
|       |               |-- dis_mips.lua
|       |               |-- dis_mipsel.lua
|       |               |-- dis_ppc.lua
|       |               |-- dis_x64.lua
|       |               |-- dis_x86.lua
|       |               |-- dump.lua
|       |               |-- p.lua
|       |               |-- v.lua
|       |               |-- vmdef.lua
|       |               `-- zone.lua
|       |-- lualib
|       |   |-- cjson.so
|       |   |-- rds
|       |   |   `-- parser.so
|       |   |-- redis
|       |   |   `-- parser.so
|       |   `-- resty
|       |       |-- aes.lua
|       |       |-- core
|       |       |   |-- base.lua
|       |       |   |-- base64.lua
|       |       |   |-- ctx.lua
|       |       |   |-- exit.lua
|       |       |   |-- hash.lua
|       |       |   |-- misc.lua
|       |       |   |-- regex.lua
|       |       |   |-- request.lua
|       |       |   |-- response.lua
|       |       |   |-- shdict.lua
|       |       |   |-- time.lua
|       |       |   |-- uri.lua
|       |       |   |-- var.lua
|       |       |   `-- worker.lua
|       |       |-- core.lua
|       |       |-- dns
|       |       |   `-- resolver.lua
|       |       |-- lock.lua
|       |       |-- lrucache
|       |       |   `-- pureffi.lua
|       |       |-- lrucache.lua
|       |       |-- md5.lua
|       |       |-- memcached.lua
|       |       |-- mysql.lua
|       |       |-- random.lua
|       |       |-- redis.lua
|       |       |-- sha.lua
|       |       |-- sha1.lua
|       |       |-- sha224.lua
|       |       |-- sha256.lua
|       |       |-- sha384.lua
|       |       |-- sha512.lua
|       |       |-- string.lua
|       |       |-- upload.lua
|       |       |-- upstream
|       |       |   `-- healthcheck.lua
|       |       `-- websocket
|       |           |-- client.lua
|       |           |-- protocol.lua
|       |           `-- server.lua
|       |-- nginx
|       |   `-- html
|       |       |-- 50x.html
|       |       `-- index.html
|       `-- sbin
|           `-- nginx
`-- usr
    `-- lib
        |-- ssl
        |   `-- openssl.cnf
        `-- x86_64-linux-gnu
            |-- libGeoIP.so.1
            |-- libcrypto.so.1.0.0
            |-- libffi.so.6
            |-- libgmp.so.10
            |-- libgnutls-deb0.so.28
            |-- libgssapi_krb5.so.2
            |-- libhogweed.so.2
            |-- libk5crypto.so.3
            |-- libkrb5.so.3
            |-- libkrb5support.so.0
            |-- liblber-2.4.so.2
            |-- libldap_r-2.4.so.2
            |-- libnettle.so.4
            |-- libp11-kit.so.0
            |-- libpq.so.5
            |-- libsasl2.so.2
            |-- libssl.so.1.0.0
            `-- libtasn1.so.6

27 directories, 99 files
14M    /tmp/resty

This whole tree is now 14MB, which is not too bad--we only have what we need, and nothing we don't.

Building the Container

Now we need to build a container. Since we mounted the docker client and a socket to the docker server, we'll build the runtime image directly from the builder:

mkdir -p /images/openresty && cp /resty.tar /images/openresty \
&& cd /images/openresty \
&& cat > Dockerfile <<EOF
FROM scratch
ADD resty.tar /
VOLUME ["/var/log/nginx","/var/cache/nginx","/var/run"]
EXPOSE 80 443
ENTRYPOINT ["/opt/openresty/sbin/nginx"]
CMD ["-V"]
EOF

cd /images/openresty \
&& docker build -t communitycloud/openresty:1.7.7.2 . \
&& docker tag communitycloud/openresty:1.7.7.2 communitycloud/openresty:latest \
&& docker push

The builder image will exit and self-destruct, leaving us with a resty image of the smallest possible size:

REPOSITORY                TAG           IMAGE ID          CREATED             VIRTUAL SIZE
communitycloud/openresty  1.7.7.2       b1c34b5b2ef8      5 seconds ago       13.43 MB

The more efficient way is to build the resty-base image with only the shared stuff from /usr directory, and then just
layer /opt with and without debug symbols. That will save us 5-6 megs. One additional way to shave a megabyte or two
would be to build against a smaller libc.

Docker Pull Command
Owner
communitycloud

Comments (0)