A basic container with
psql, the PostgreSQL Command Line Client installed.
This project includes a
Makefile with targets to build & ship the container.
make build # to build make ship # to build & push to Docker Hub (you must be a maintainer for permissions)
To run the pod in Docker:
docker run --rm -ti \ -e PGUSER=your_db_user \ -e PGPASSWORD=your_db_password \ -e PGDATABASE=your_db_name \ -e PGHOST=your_db_host \ -e PGPORT=5432 \ returnpath/psql -c '\dS'
To run a test pod in Kubernetes via a ReplicationController, first create a Secret containing your
base64 encoded database credentials:
apiVersion: v1 kind: Secret metadata: name: db-secret-name labels: environment: prod type: Opaque data: pguser: <PGUSER> pgpassword: <PGPASSWORD> pgdatabase: <PGDATABASE> pghost: <PGHOST> pgport: <PGPORT>
To encode your credentials:
echo -n 'your-pg-user' | base64
- Paste in the result into the
db-secret.ymlin place of
- Repeat for all the other variables
- Create the secret into the Kubernetes cluster:
kubectl apply -f db-secret.yml
Then, create a file
apiVersion: v1 kind: ReplicationController metadata: name: psql spec: replicas: 1 selector: name: psql-test template: metadata: labels: name: psql-test service: psql-test spec: containers: - name: psql-test image: returnpath/psql env: - name: PGUSER valueFrom: secretKeyRef: name: db-secret-name key: pguser - name: PGPASSWORD valueFrom: secretKeyRef: name: db-secret-name key: pgpassword - name: PGDATABASE valueFrom: secretKeyRef: name: db-secret-name key: pgdatabase - name: PGHOST valueFrom: secretKeyRef: name: db-secret-name key: pghost - name: PGPORT valueFrom: secretKeyRef: name: db-secret-name key: pgport args: - "-c" - "\\dS" restartPolicy: Always
kubectl apply -f psql-rc-test.yaml
A Pod & ReplicationController should start and run the command:
psql -c \dS
The above example is only truly useful as a test for whether this pod & your database connection is working. In order to make practical use of this container, you probably want to run it with queries as scheduled jobs.
ScheduledJob construct did not exist. If you are using a version of Kubernetes without
ScheduledJob, you probably want to check out: ReturnPath/job-runner.
In order to use the job runner, you should use the
kubernetes-native Job Runner. The job-runner's
ONBUILD instructions will copy
k8s-jobs directories into your image which is based on it.
First, create a Docker image based on
returnpath/job-runner to contain your job definitions:
Create a directory
k8s-jobs/, then place native Kubernetes Job
.yaml files in it. An example for using this
psql container might be:
apiVersion: batch/v1 kind: Job metadata: name: psql-test spec: template: metadata: name: psql-test labels: service: psql-test spec: containers: - name: psql-test image: returnpath/psql args: [ "-c", "\\dS" ] env: - name: PGUSER valueFrom: secretKeyRef: name: db-secret-name key: pguser - name: PGPASSWORD valueFrom: secretKeyRef: name: db-secret-name key: pgpassword - name: PGDATABASE valueFrom: secretKeyRef: name: db-secret-name key: pgdatabase - name: PGHOST valueFrom: secretKeyRef: name: db-secret-name key: pghost - name: PGPORT valueFrom: secretKeyRef: name: db-secret-name key: pgport restartPolicy: Never
Then, create a
crond style schedule for the job by creating a directory named
cron/ with a file with the same name as your job, passing your job name as the first argument to
/app/processor/runner. For example, to run it every
mkdir -p cron/ echo "*/5 * * * * root /app/processor/runner psql-test >> /var/log/cron.log 2>&1" > cron/psql-test
Then, create a
defaults/job-name file, even if it is blank! It is important to note that without this file, the job reaper will not delete the job, and subsequent runs will fail due to already having existed. This can be useful for testing that a job runs one time & inspecting things afterwards without worrying about the reaper deleting jobs & pods.
Any files under
default will be copied into
/etc/default/, and sourced in the context of the job-runner container's "runner" script. This can be useful for setting environment variables or executing pre-task bash commands or functions.
To create a blank
/etc/defaults file for the job:
Next, build your job container with:
docker build -t your-dockerhub-username/test-jobs . docker push your-dockerhub-username/test-jobs
Finally, start the job-runner container you have just built on the cluster. For example, to run
1 instance of the job-runner via a ReplicationController, create a file:
apiVersion: v1 kind: ReplicationController metadata: name: job-runner spec: replicas: 1 selector: name: job-runner template: metadata: labels: name: job-runner spec: containers: - name: job-runner image: your-dockerhub-username/test-jobs env: - name: RUNNER value: kubernetes-native - name: KUBERNETES_MASTER value: https://kubernetes:443 kubectl apply -f job-runner.yaml
Note: If you wish to run multiple instances of the
job-runner, you will need to use the "Distributed Locking" feature of the base container. This involves installing Consul, and is a bit more complicated than this README will delve into.
If all went well, you should now have a test job scheduled to run every 5 minutes. To see the jobs & pods:
kubectl get pods -l name=job-runner kubectl describe rc,pod -l name=job-runner kubectl get jobs -l service=psql-test kubectl get pods -l service=psql-test kubectl describe jobs,pods -l service=psql-test
To follow the logs for the job-runner or job pods, find the unique pod name for each via the above commands and run either of:
kubectl logs -f job-runner-xxxxx kubectl logs -f psql-test-xxxxx
This project is simply a packaging script for the
awscli tools. As such, nothing in this repository is "novel", or "non-obvious". This repo is therefore released under the permissive MIT License.
However, the upstream tools are released under various Open Source Licenses:
The text of these tool's licenses are included here to avoid confusion.