sptrakesh/s3-proxy

docker

Hub

By sptrakesh

•Updated 6 days ago

Simple proxy server based on nghttp2 or Boost:Beast to serve files stored in an AWS S3 bucket.

Image

Web Servers

5.9K

Overview Tags

Simple proxy server⁠ built using Boost:Beast and the AWS C++ SDK for serving files stored in AWS S3 buckets. Useful in cases where CloudFront does not work (when using higher strength SSL certificates). Can be used to serve content from private buckets since the container needs to be configured with a valid AWS key-pair (suggest a restricted account with only read-only privilege to S3).

Example of running

docker run -d --rm \
  -p 8000:8000 \
  -e AWS_REGION="<aws region>" \
  -e S3_BUCKET="<your bucket>" \
  -e AWS_KEY="<restricted account Access key ID>" \
  -e AWS_SECRET="<restricted account Secret access key>" \
  -e AUTH_KEY="<internal secret key to protect cache clear endpoint>" \
  --name s3-proxy sptrakesh/s3-proxy

Configuration

Additional environment variables are supported to configure the container:

TTL - TTL for the local cache. The container downloads and caches objects retrieved from S3 for the specified time (in seconds). Default 300 (5 minutes).
CACHE_DIR - Directory to use to cache the S3 objects. Default /opt/spt/data. Specify a path to a volume or local mount as appropriate.
PORT - Port on which the service is to be run.
THREADS - The number of threads to spawn. Default 8.
AUTH_KEY - The internal secret key to protect the cache clear endpoint.

Optional Bindings

The service may optionally be configured to publish metrics (requests, errors, potential region from which request originated etc) to Akumuli TSDB via the mmdb-ws service. This can be enabled via the following additional options. Note that both mmdb and akumuli need to configured together.

MMDB_HOST - The host on which the mmdb-ws runs. Integration will be disabled if not specified.
MMDB_PORT - The port on which the mmdb-ws service runs. Default 8010.
AKUMULI_HOST - The host on which Akumuli TSDB runs. Integration will be disabled if not specified.
AKUMULI_PORT - The port on which Akumuli TCP write socket listens. Default 8282.
METRIC_PREFIX - A prefix to associate with the metric names. Default request. Note: all metrics will use <prefix>. as the prefix.

Stack

The following illustrates a docker stack running the proxy server with metrics being published to Akumuli via mmdb.

Docker stack file

Sample docker compose file to run the stack.

version: '3.7'

services:
  akumuli:
    image: akumuli/akumuli:0.8.80-skylake
    environment:
      - nvolumes=0
    stop_signal: SIGTERM
    ports:
      - "8181:8181"
      - "8282:8282"
      - "8383:8383"
    volumes:
      - $DATA_DIR/Akumuli:/Akumuli

  mmdb-ws:
    image: sptrakesh/mmdb-ws
    stop_signal: SIGTERM
    ports:
      - "8010:8010"
    volumes:
      - $DATA_DIR/dbip-city-lite-2020-05.mmdb.gz:/opt/spt/data/dbip.mmdb.gz:ro

  s3-proxy:
    image: sptrakesh/s3-proxy
    stop_signal: SIGTERM
    ports:
      - "8000:8000"
    environment:
      - AWS_REGION=<aws region>
      - S3_BUCKET=<aws s3 bucket>
      - AWS_KEY=<aws key>
      - AWS_SECRET=<aws secret key>
      - AUTH_KEY=<cache clear auth key>
      - MMDB_HOST=mmdb-ws
      - AKUMULI_HOST=akumuli
    volumes:
      - $DATA_DIR/s3-proxy:/opt/spt/logs

  geojson-ds:
    image: sptrakesh/geojson-ds
    stop_signal: SIGTERM
    ports:
      - "8020:8020"
    environment:
      - PORT=8020
      - THREADS=2
      - AKUMULI_HOST=akumuli
      - AKUMULI_PORT=8181

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana:/var/lib/grafana
      - $DATA_DIR/grafana/akumuli-datasource:/var/lib/grafana/plugins/akumuli-datasource
      - $DATA_DIR/grafana/alexandra-trackmap-panel:/var/lib/grafana/plugins/alexandra-trackmap-panel
      - $DATA_DIR/grafana/simple-json-datasource:/var/lib/grafana/plugins/simple-json-datasource

volumes:
  grafana:

Shell Script

The following illustrates a shell script to run the stack.

#!/bin/sh

STACK=spt
DATA_DIR=$SPT_DATA_DIR
if [ -z "$DATA_DIR" ]
then
  DATA_DIR='/private/var/tmp/spt'
fi

Directory() {
  if [ ! -d $1 ]
  then
    mkdir -p $1
  fi
  chmod 777 $1
}

Swarm()
{
  running=`docker system info | grep Swarm | awk '{print $2}'`
  if [ "$running" == "inactive" ]
  then
    docker swarm init
  fi
}

Grafana()
{
  docker pull grafana/grafana
  GROOT=$DATA_DIR/grafana
  Directory $GROOT

  akplugin=$GROOT/akumuli-datasource
  if [ ! -d $akplugin ]
  then
    (cd $GROOT; git clone https://github.com/akumuli/akumuli-datasource.git)
  else
    (cd $akplugin; git pull)
  fi

  mplugin=$GROOT/alexandra-trackmap-panel
  if [ ! -d $mplugin ]
  then
    (cd $GROOT; git clone https://github.com/alexandrainst/alexandra-trackmap-panel.git)
  else
    (cd $mplugin; git pull)
  fi

  jplugin=$GROOT/simple-json-datasource
  if [ ! -d $jplugin ]
  then
    (cd $GROOT; git clone https://github.com/grafana/simple-json-datasource.git)
  else
    (cd $jplugin; git pull)
  fi
}

Stack()
{
  Swarm
  docker pull akumuli/akumuli:0.8.80-skylake
  Directory $DATA_DIR/Akumuli
  Directory $DATA_DIR/s3-proxy
  Grafana
  (cd `dirname $0`;
    DATA_DIR=$DATA_DIR docker stack deploy -c stack.yml $STACK)
}

Logs()
{
  if [ -z "$2" ]
  then
    for i in `docker stack ps $STACK | grep $1 | grep Running | awk '{print $1}'`
    do
      echo "Logs for $i"
      echo ""
      echo ""
      docker service logs "$i"
      echo ""
      echo ""
      echo ""
    done
  else
    for i in `docker stack ps $STACK | grep $1 | grep Running | awk '{print $1}'`
    do
      echo "Logs for $i"
      echo ""
      echo ""
      docker service logs "$i" | tail "-$2"
      echo ""
      echo ""
      echo ""
    done
  fi
}

case "$1" in
  'start')
    Stack
    ;;
  'logs')
    if [ -z "$2" ]
    then
      echo "Usage: $0 logs <service> [number of lines]"
      exit 2
    else
      Logs $2 $3
    fi
    ;;
  'status')
    docker service ls
    docker stack ps $STACK
    ;;
  'update')
    if [ -z "$2" ]
    then
      docker service update --force $STACK"_"s3-proxy
    else
      docker service update --force $STACK"_$2"
    fi
    ;;
  'stop')
    docker stack rm $STACK
    ;;
  *)
    echo "Usage: $0 <start|logs|status|update|stop>"
    ;;
esac

Docker Pull Command

docker pull sptrakesh/s3-proxy