openbridge/nginx repository overview

⁠NGINX Accelerated including Reverse Proxy, Redis, CDN, Lets Encrypt and much more!

This is a Docker image creates a high performance, optimized image for NGINX. Deliver sites and applications with performance, reliability, security, and scale. This NGINX server offers advanced performance, web and mobile acceleration, security controls, application monitoring, and management.

⁠Features

The image includes configuration enhancements for;

• Reverse Proxy
• SEO optimizations
• Customizable configurations
• SSL with support for Lets Encrypt SSL certificates
• Mime-type based caching
• Redis LRU cache
• Fastcgi cache
• Proxy cache
• tmpfs file cache
• Brotli and Gzip compression
• Redirects for moved content
• Security & Bot Protection⁠
• Monitoring processes, ports, permissions... with Monit
• Standardized UID/GID and Permissions (www-data)
• Support GeoIP
• Rate limited connections to slow down attackers
• CDN support
• Cache purge
• Pair with high performance PHP-FPM container⁠

There are many, many other benefits to this system. Give it a try!

⁠Getting Started

⁠Install

The first step is to build or pull the image:

⁠Build

docker build --build-arg "NGINX_VERSION=1.15.4" -t openbridge/nginx .

Replace NGINX_VERSION=1.15.4 with your preferred version. You can aslo simply pull the images. See below.

⁠Pull

docker pull openbridge/nginx:latest

You can also use a different version of NGINX simply by pulling a build with the NGINX version you want. For example;

docker pull openbridge/nginx:1.15.3
docker pull openbridge/nginx:1.15.2
docker pull openbridge/nginx:1.15.1

To see the available versions vist https://hub.docker.com/r/openbridge/nginx/tags/⁠

⁠Running

Via Docker compose

docker-compose up -d

Using a named yml file:

docker-compose -f ./compose/html.yml up -d --remove-orphans

There are a sample HTML compose file at ./compose/html.yml as well as PHP one ./compose/php.yml to get you started.

⁠Understanding Configurations

Please note that the config files included in /conf/* directory are opinionated. They are working examples of a specific implementation preferences and needs.

We have provided two pre-built configurations. The first is for html based sites within /conf/html/* and the other is for php sites within /conf/php/*. The application will look for a config directory to use. This is done with the NGINX_CONFIG ENV variable. For example, if you are running a html based site and want to use the /conf/html/* configuration then set NGINX_CONFIG=html. If you want are running php and want to use the /conf/php/* configuration then set NGINX_CONFIG=php. If you have a custom config set like /conf/<my-custom-config>/* then set NGINX_CONFIG=my-custom-config.

There is nginx default setup located here /conf/basic/*. Basic allows you to run nginx in a bare metal setup. Just set NGINX_CONFIG=basic

⁠Digging Into The ENV File

The following are the core variables for the ENV config in /conf

• NGINX_DOCROOT sets the default www directory. If you do not set this the images defaults to /usr/share/nginx/html
• NGINX_SERVER_NAME sets the default servern ame in nginx.conf. If you do not set this it will default to localhost
• NGINX_CONFIG sets the default configuration director for your image. See the /conf directory to review a html and php configuration
• NGINX_PROXY_UPSTREAM sets the upstream server(s) for the reverse proxy to connect with. Since the proxy is local to the container you should use something like localhost.com:8080. If this is NOT set, it will default to localhost:8080
• REDIS_UPSTREAM sets the upstream Redis cache server(s) to connect with. If you are using compose you might reference the redis container server redis01:6379;server redis02:6378;. You might also set it by IP server 1.2.3.4:6379; server 4.3.2.1:6379;. If this is NOT set, it will default to server localhost:6379;.

If you are using PHP you will want to set the endpoint for PHP-FPM:

• PHP_FPM_UPSTREAM sets the upstream server(s) to connect with. If you are using compose you might reference the php-fpm01 container server php-fpm01:9000;server php-fpm01:9001;. You might also set it by IP server 1.2.3.4:9000; server 4.3.2.1:9001;. If this is NOT set, it will default to server localhost:9000;

You can set a collection of dummy files and certs for local testing:

• NGINX_DEV_INSTALL Set to true if you want self-signed SSL certs installed and "hello world" HTML and PHP pages installed. This is useful for testing.
• NOTE: Self-signed SSL certificates are always installed if the system does not detect here in the default cert location at /etc/letsencrypt/live/${NGINX_SERVER_NAME}/

Check our the /env for more examples

⁠Mounting Your Web App or Site Content

Following is the convention we will be using for sites.

• /etc/nginx/sites-available/ – nginx configuration for different sites will be available
• /usr/share/nginx/html – Your root site content/apps
• /usr/share/nginx/html/example.com – (Optional) Domain specific content/apps

To mount your web app or html files you will need to mount the volume on the host that contains your files. Make sure you are setting the NGINX_DOCROOT in your run or docker-compose.yml file. If you do not set it the default is /usr/share/nginx/html

-v /your/webapp/path:{{NGINX_DOCROOT}}:ro

You can also set the cache directory to leverage in-memory cache like tmpfs:

-v /tmpfs:{{CACHE_PREFIX}}:ro

You can do the same thing for config files if you wanted to use versions of what we have provided. Just make sure you are mapping locations correctly as NGINX and PHP expect files to be in certain locations.

⁠NGINX /conf/ Configuration File Organization

The following represents the structure of the configs used in this image. Please note the use of the nginx map settings for browser content caching and supporting content redirects. The content cache is done according to mime type.

⁠NGINX Configuration Files:

• /etc/nginx/ – all nginx related configuration will be in this folder
• /etc/nginx/bots.d/* – Bot and spam configuration file
• /etc/nginx/conf.d/* – Core configuration files like PHP, security, SEO...
• /etc/nginx/fastcgi.d/* – The base fastcgi configuration
• /etc/nginx/geo.d/* – Configuration files for IP allow/deny
• /etc/nginx/header.d/* – The httpd, proxy and fastcgi headers
• /etc/nginx/map.d/* – Configuration files for caching, access rules, redirects....
• /etc/nginx/redis.d/* – Configuration files for Redis caching
• /etc/nginx/site-available/* – The vhost configurations
• /etc/nginx/status.d/* – Configuration to allow access to nginx server stats
• /etc/nginx/upstream.d/* – The upstream server(s)
• /etc/nginx/nginx.conf – The parent nginx configuration file

Here is a general layout for the configs though this may change over time:

/etc/nginx/
├── nginx.conf
├── conf.d/
│   ├── gzip.conf
│   ├── ssl.conf
│   ├── secure.conf
│   ├── location.conf
│   ├── seo.conf
│   └── nocache.conf
├── map.d
│   ├── cache/
│   │   ├── expires.map
│   └── redirects/
│       ├── example.map
│       ├── pages.map
├── bot.d
├── fastcgi.d
├── header.d
├── geo.d
├── redis.d
├── status.d
├── upstream.d
├── sites-available

⁠.nginx.conf Configuration

This is base contents of the configuration file (see actual nginx.conf for current config).

⁠conf.d and map.d Configuration Files

It is common practice to partition config files into a conf.d directory. This is no different. However, what is different is the use of the ngx_http_map_module⁠ for configuration. There are a few different map use cases represented. The use of map is preferred over if conditional statements and follows Nginx best practices.

⁠Example HTML configuration conf/html/conf.d/location.conf

Below is an example of an advanced configuration that is designed to support single page applications (Angular, React...) based on a set of rules for bots and crawlers. If you are not using a prerender service, comment out those details for serving basic html:

location / {
   try_files                   $uri /$uri;
   aio                         threads;
   include                     /etc/nginx/redis.d/cache.conf;
}

⁠Access (map.d/access/*)

Using map we set access policies based on address

0 $binary_remote_addr;
1 "";

⁠Browser Cache (map.d/cache/*)

Using map we set the browser cache expires settings by mime-type. For example, here are some mime mappings with the expires policy:

image/gif max;
image/jpeg max;
image/png max;
image/svg+xml max;

⁠Headers (map.d/header/*)

Use map to set conditional headers based on sets of rules which is useful for SEO, schemes and other purposes

⁠Logs (map.d/logs/*)

Sets a conditional so we are not logging internal traffic. For example, we don't want Monit health check requests showing up in the logs.

⁠No Cache (map.d/nocache/*)

Sets conditions for resources we do not want to have cached. For example, /wp-admin/ for a Wordpress blog.

⁠Cache Purge (map.d/purge/*)

Supports requests for conditional cache purges using ngx_cache_purge⁠.

⁠Redirects (map.d/purge/*)

The second is using map for redirects for pages that are missing or have moved. For example, here is a map for moving the blog location:

/blog/ https://blog.openbridge.com;

⁠Referrers (map.d/referrer/*)

Set conditional behaviors based on referrer, including detecting bots.

⁠srcache (map.d/srcache/*)

The ngx_srcache⁠ is used for Redis LRU cache. It provides transparent subrequest-based caching layout for arbitrary nginx locations.

⁠PHP-FPM

These are the primary php-fpm configurations in /conf/php/nginx:

/conf.d/location.conf
/fastcgi.d/fastcgi.conf

Please note that PHP caching is active and using TCP (vs sockets)

Check out the high performance docker PHP service we published here: openbridge/ob_php-fpm⁠

Server locations are define via upstream:

upstream php-fpm {include /etc/nginx/upstream.d/php-fpm.conf;}

While you can pair this with any php-fpm service, it is recommended that you use this highly optimized PHP-FPM Docker Service⁠. Most of our docker-compose files will reference this.

⁠Redis Cache

The image is setup to use Redis as a reverse proxy LRU cache. The current cache settings reflect a balance of performance, optimization and striving for efficiencies with little resource consumption as possible. There are three main configs for Redis:

/redis.d/location.conf
/redis.d/cache.conf
/upstream.d/redis.conf

⁠What is a Redis LRU cache?

When an http client requests a web page Nginx looks for the corresponding cached object in Redis. If the object is in redis, nginx serves it. If the object is not in Redis, Nginx requests a backend that generates the page and gives it back to Nginx. Then, Nginx put it in Redis and serves it. All cached objects in Redis have a configurable TTL with a default of 15s.

⁠SSL

To keep things organized we default to using letsencrypt⁠ for SSL certificates/keys, paths and naming conventions. Your are free to use something other than letsencrypt, just follow the path and naming conventions set forth below. In keeping with the letsencrypt conventions make sure your certs are using the same naming scheme:

/etc/letsencrypt/live/${NGINX_SERVER_NAME}/;
├── server
│   ├── cert.pem
│   ├── chain.pem
│   ├── fullchain.pem
│   └── privkey.pem

The default locations for the SSL certs are in /conf.d/ssl.conf:

ssl_certificate /etc/letsencrypt/live/{{NGINX_SERVER_NAME}}/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/{{NGINX_SERVER_NAME}}/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/{{NGINX_SERVER_NAME}}/chain.pem;

⁠Mount Your SSL Certs

You mount your certs directory on the host to the certs: /etc/letsencrypt/live/${NGINX_SERVER_NAME}.

-v /your/certs/path:/etc/letsencrypt/live/{{NGINX_SERVER_NAME}}:ro

In the event you are not using letsencrypt, mount your local SSL files in the same manner:

- /path/to/ssl/www.domain.com/fullchain.pem:/etc/letsencrypt/live/www.domain.com/fullchain.pem
- /path/to/ssl/www.domain.com/privkey.pem:/etc/letsencrypt/live/www.domain.com/privkey.pem
- /path/to/ssl/www.domain.com/chain.pem:/etc/letsencrypt/live/www.domain.com/chain.pem

The following is an example docker compose file that shows how to mount SSL certs from the host into your container with the correct pathing:

version: '3.1'
services:
  nginx:
    image: openbridge/nginx:latest
    container_name: nginx
    depends_on:
      - redis
    ports:
      - 80:80
      - 443:443
    tty: true
    restart: always
    tmpfs: /var/cache
    volumes:
      - /path/user/html:/usr/share/nginx/html
      - /etc/letsencrypt/live/www.domain.com/fullchain.pem:/etc/letsencrypt/live/www.domain.com/fullchain.pem
      - /etc/letsencrypt/live/www.domain.com/privkey.pem:/etc/letsencrypt/live/www.domain.com/privkey.pem
      - /etc/letsencrypt/live/www.domain.com/chain.pem:/etc/letsencrypt/live/www.domain.com/chain.pem
    ulimits:
      nproc: 65535
      nofile:
          soft: 49999
          hard: 99999
    env_file:
        - ./env/prod.env
  redis:
    image: redis:alpine
    container_name: redis
    restart: always
  volumes:
    site:

⁠Installing certbot for letsencrypt SSL certs

On your host, not in the Docker image, install certbot:

• Download certbot: curl -O https://dl.eff.org/certbot-auto
• Set permissions: chmod +x certbot-auto
• Move the executable: mv certbot-auto /usr/local/bin/certbot-auto
• Generate your certificate: /usr/local/bin/certbot-auto certonly -n --debug --agree-tos --email [email protected] --standalone -d *.openbridge.com

If your run into an errors with certbot, trying running these commands:

rm -rf ~/.local/share/letsencrypt
rm -rf /opt/eff.org/*
pip install -U certbot
#try this
certbot certonly -n --debug --agree-tos --pre-hook="docker stop nginx" --post-hook="docker start nginx" --standalone -d {{yourdomain.com}} > /dev/null
# or this
certbot renew --debug

Certbot seems to be sensitive to OS and python updates and removing these files has helped clear up issues in the past.

You will need to setup a renewal process. The docs say check twice a day for changes. Lets add the renewal process to cron:

cat << EOF > /tmp/crontab.conf
55 4,16 * * * /opt/eff.org/certbot/venv/local/bin/pip install --upgrade certbot
59 4,16 * * * /usr/local/bin/certbot certonly -n --debug --agree-tos --pre-hook="docker stop nginx" --post-hook="docker start nginx" --standalone -d *.openbridge.com > /dev/null
EOF

Lastly, add everything to cron via cat /tmp/crontab.conf | crontab - && crontab -l

⁠Local Development SSL Certs

If you set NGINX_DEV_INSTALL=true it will install a self-signed SSL certs for you. If you already have mounted dev certs, it will not install them as it assumes you want to use those. Here is the code that does this when you set set NGINX_DEV_INSTALL=true:

if [[ ! -f /etc/letsencrypt/live/${NGINX_SERVER_NAME}/privkey.pem ]] || [[ ! -f /etc/letsencrypt/live/${NGINX_SERVER_NAME}/fullchain.pem ]]; then

  echo "OK: Installing development SSL certificates..."
  mkdir -p /etc/letsencrypt/live/${NGINX_SERVER_NAME}

  /usr/bin/env bash -c "openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -subj /C=US/ST=MA/L=Boston/O=ACMECORP/CN=${NGINX_SERVER_NAME} -keyout /etc/letsencrypt/live/${NGINX_SERVER_NAME}/privkey.pem -out /etc/letsencrypt/live/${NGINX_SERVER_NAME}/fullchain.pem"

  cp /etc/letsencrypt/live/${NGINX_SERVER_NAME}/fullchain.pem  /etc/letsencrypt/live/${NGINX_SERVER_NAME}/chain.pem

else
  echo "INFO: SSL files already exist. Not installing dev certs."
fi

⁠Forward Secrecy & Diffie Hellman Ephemeral Parameters

The default Ephemeral Diffie-Hellman (DHE) uses OpenSSL's defaults, which include a 1024-bit key for the key-exchange. Since we're using a 2048-bit certificate, DHE clients will use a weaker key-exchange than non-ephemeral DH clients. We need to fix this. We generate a stronger DHE parameter which can take a LONG time to generate:

openssl dhparam -out dhparam.pem 2048

BE PATIENT!

If you have already generated this, mount it to /etc/pki/tls/dhparam.pem and the container will forgo creating it.

⁠HTTP Strict Transport Security

We have enabled HTTP Strict Transport Security (HSTS), which instructs browsers to communicate only over HTTPS.

⁠Qualsys Rating

Using the Qualsys SSL Test (https://www.ssllabs.com/ssltest/⁠) the current SSL configuration scores A+

⁠Permissions

We have standardized on the user, group and UID/GID to work seamlessly with other applications like PHP-FPM⁠

&& addgroup -g 82 -S www-data \
&& adduser -u 82 -D -S -h /var/cache/php-fpm -s /sbin/nologin -G www-data www-data \

⁠Bots & Spam Protection

We have include "The Ultimate Nginx Bad Bot, User-Agent, Spam Referrer Blocker, Adware, Malware and Ransomware Blocker, Clickjacking Blocker, Click Re-Directing Blocker, SEO Companies and Bad IP Blocker with Anti DDOS System, Nginx Rate Limiting and Wordpress Theme Detector Blocking"

⁠What Are Bad Bots?

Why block bad bots? They can cause problems for your application, performance and allow pollute your performance data! What are some examples of bad bots and spam? Bots attempt to make themselves look like other software or web sites by disguising their user agent. Their user agent names may look harmless, perfectly legitimate even.

• Bad Referrers
• Bad User-Agent Strings
• Spam Referrers
• Spam Bots and Bad Bots
• Sites Linked to Lucrative Malware, Adware and Ransomware Clickjacking Campaigns
• Vulnerability scanners
• Gambling and Porn Web Sites
• E-mail harvesters
• Content scrapers
• Aggressive bots that scrape content
• Image Hotlinking Sites and Image Thieves
• Bots or Servers linked to viruses or malware
• Government surveillance bots
• Botnet Attack Networks (Mirai)
• Known Wordpress Theme Detectors (Updated Regularly)
• SEO companies that your competitors use to try improve their SEO
• Link Research and Backlink Testing Tools
• Stopping Google Analytics Ghost Spam
• Browser Adware and Malware (Yontoo etc)

There are over 4000 bad referers, spam referrers, user-agents, bad bots, bad IP's, porn, gambling and clickjacking sites, lucrative seo companies, wordpress theme detectors and so forth.

⁠Beneifts

• Extensive Lists of Bad and Known Bad Bots and Scrapers (updated almost daily)
• Blocking of Spam Referrer Domains and Web Sites
• Blocking of SEO data collection companies like Semalt.com, Builtwith.com, WooRank.com and many others (updated regularly)
• Blocking of clickjacking Sites linked to Adware, Malware and Ransomware
• Blocking of Porn and Gambling Web Sites who use Lucrative Ways to Earn Money through Serving Ads by hopping off your domain names and web sites.
• Blocking of Bad Domains and IP's that you cannot even see in your Nginx Logs. Thanks to the Content Security Policy (CSP) on all my SSL sites I can see things trying to pull resources off my sites before they even get to Nginx and get blocked by the CSP.
• Anti DDOS Filter and Rate Limiting of Agressive Bots
• Alphabetically ordered for easier maintenance (Pull Requests Welcomed)
• Commented sections of certain important bots to be sure of before blocking
• Includes the IP range of Cyveillance who are known to ignore robots.txt rules and snoop around all over the Internet.
• Whitelisting of Google, Bing and Cloudflare IP Ranges
• Whitelisting of your own IP Ranges that you want to avoid blocking by mistake.
• Ability to add other IP ranges and IP blocks that you want to block out.
• If its out there and it's bad it's already in here and BLOCKED !!

⁠Verify Your Bot Protection

Run the following commands one by one from a terminal on another linux machine against your own domain name.

Substitute yourdomain.com in the examples below with your REAL domain name:

curl -A "googlebot" http://yourdomain.com

Should respond with 200 OK

curl -A "80legs" http://yourdomain.com

curl -A "masscan" http://yourdomain.com

Should respond with: curl: (52) Empty reply from server

curl -I http://yourdomain.com -e http://100dollars-seo.com

curl -I http://yourdomain.com -e http://zx6.ru

Should respond with: curl: (52) Empty reply from server

The Bot Blocker is now WORKING and PROTECTING your sites!

⁠Testing

⁠HTML & PHP

If you set NGINX_DEV_INSTALL=true it will install a self-signed SSL cert AND copy test files to a test directory:

cp /tmp/index.php "${NGINX_DOCROOT}"/testing/test_info.php
cp /tmp/test.html "${NGINX_DOCROOT}"/testing/test_nginx.html

This allows you to connect to https://localhost/testing/test_info.php to verify the container is working correctly. If you connect to https://localhost/testing/test_nginx.html it will show a hello world test page.

Noe: Using PHP assumes you have configured a PHP backend to test anything PHP related

⁠Monitoring

Services in the container are monitored via Monit. One thing to note is that if Monit detects a problem with Nginx it will issue a STOP command. This will shutdown your container because the image uses CMD ["nginx", "-g", "daemon off;"]. If you are using --restart unless-stopped in your docker run command the server will automatically restart.

Here is an example monitoring config:

check process nginx with pidfile "/var/run/nginx.pid"
      if not exist for 5 cycles then restart
      start program = "/usr/bin/env bash -c '/usr/sbin/nginx -g daemon off'" with timeout 60 seconds
      stop program = "/usr/bin/env bash -c '/usr/sbin/nginx -s stop'"
      every 3 cycles
      if cpu > 80% for 10 cycles then exec "/usr/bin/env bash -c '/usr/sbin/nginx -s stop'"

check program wwwdata-permissions with path /usr/bin/env bash -c "check_wwwdata permission"
      every 3 cycles
      if status != 0 then exec "/usr/bin/env bash -c 'find {{NGINX_DOCROOT}} -type d -exec chmod 755 {} \; && find {{NGINX_DOCROOT}} -type f -exec chmod 644 {} \;'"

check directory cache-permissions with path {{CACHE_PREFIX}}
      every 3 cycles
      if failed permission 755 then exec "/usr/bin/env bash -c 'find {{CACHE_PREFIX}} -type d -exec chmod 755 {} \;'"

check directory cache-owner with path {{CACHE_PREFIX}}
      every 3 cycles
      if failed uid www-data then exec "/usr/bin/env bash -c 'find {{CACHE_PREFIX}} -type d -exec chown www-data:www-data {} \; && find {{CACHE_PREFIX}} -type f -exec chown www-data:www-data {} \;'"

check file letsencrypt_certificate with path /etc/letsencrypt/live/{{NGINX_SERVER_NAME}}/fullchain.pem
      if changed checksum then exec "/usr/bin/env bash -c '/usr/sbin/nginx -s reload'"

check host {{NGINX_SERVER_NAME}} with address {{NGINX_SERVER_NAME}}
      if failed host {{NGINX_SERVER_NAME}} port 80 protocol http
        and request "/health-check"
        with timeout 25 seconds
        for 3 times within 4 cycles
        then exec "/usr/bin/env bash -c '/usr/sbin/nginx -s reload'"
      if failed host {{NGINX_SERVER_NAME}} port 443 protocol https
        request "/health-check"
        status = 200
        content = "healthy"
        with timeout 25 seconds
        for 3 times within 4 cycles
      then exec "/usr/bin/env bash -c '/usr/sbin/nginx -s reload'"
      if failed port 8080 for 3 cycles then exec "/usr/bin/env bash -c '/usr/sbin/nginx -s stop'"

check program cache-size with path /usr/bin/env bash -c "check_folder {{CACHE_PREFIX}} 500"
      every 20 cycles
      if status != 0 then exec "/usr/bin/env bash -c 'rm -Rf /var/cache/*'"

The check_folder, check_host and check_wwwdata scripts provide additional health check utility of make sure that permissions, cache size and host respond correctly. For example, check_host will validate that SPA rendering service is properly serving the expected content. This can help detect if there are issues where certain user-agents that can not render SPA are being served the incorrect content. This can wreak havoc with your SEO