jonhartley/kubernetes-pxc

By jonhartley

•Updated about 7 years ago

CentOS 7 / Percona XtraDB 5.7 Container specifically for Kubernetes StatefulSet

Image

10K+

Overview Tags

jonhartley/kubernetes-pxc repository overview

⁠Percona 5.7 XtraDB Cluster

⁠Basic Info

This is Percona's version of Galera cluster for MySQL in Docker or Kubernetes
Includes the ability to start from a backup, autotune config, be a slave cluster and SSL config

⁠ENV Variables

CONFIG="max_allowed_packet=1G,innodb_buffer_pool_size=500M,max_connections=1000"
WSREPCONFIG="wsrep_slave_threads=4"
MYSQL_ROOT_PASSWORD=password -- root password
SSTUSER_PASSWORD=password -- sstuser password
CLUSTER_NAME=pxc_cluster -- WSREP cluster name
BINLOG="true" -- create a binlog
SLAVE="true" -- bw a slave of another cluster
REPLUSER="replication:password" -- replication credentials
MASTER_HOST="pxc.srv.kubernetes" -- where your master is
STARTFROMBACKUP="true" -- restore backup before starting
BACKUPDIR="/var/backup" -- location of latest backup
CPU_LIMIT -- use Downward API to push limits.cpu
MEM_LIMIT -- use Downward API to push limits.memory
AUTOTUNE="connections|bufferpool" -- tune configuration based on CPU_LIMIT | MEM_LIMIT
SSL="client|all" -- both option require ssl-cert,ssl-key and ssl-ca in CONFIG and certs mounted on container
PROXYMON="proxymon:password" -- basic user to monitor cluster (granted only USAGE)
It should run in Docker so you can test config - use naming conventions as if it was in a Kubernetes Stateful Set - use SSSVC to bootstrap it from container 1

docker network create pxccluster


docker run --net pxccluster \
--name pxc-0 \
--hostname pxc-0 \
-v /tmp/pxc1:/var/lib/mysql \
-e CONFIG="max_allowed_packet=1G,innodb_buffer_pool_size=500M,max_connections=1000,wsrep_slave_threads=4" \
-e MYSQL_ROOT_PASSWORD=password \
-e SSTUSER_PASSWORD=password \
-e CLUSTER_NAME=pxc_cluster \
-e CPU_LIMIT=2 \
-e MEM_LIMIT=2147483648 \
-e AUTOTUNE=bufferpool \
-e SSSVC=pxc-0 \
-e NODE_NAME=pxc-0 \
-d jonhartley/kubernetes-pxc

docker run --net pxccluster \
--name pxc-1 \
--hostname pxc-1 \
-v /tmp/pxc2:/var/lib/mysql \
-e CONFIG="max_allowed_packet=1G,innodb_buffer_pool_size=500M,max_connections=1000,wsrep_slave_threads=4" \
-e MYSQL_ROOT_PASSWORD=password \
-e SSTUSER_PASSWORD=password \
-e CLUSTER_NAME=pxc_cluster \
-e CPU_LIMIT=2 \
-e MEM_LIMIT=2147483648 \
-e AUTOTUNE=bufferpool \
-e SSSVC=pxc-0 \
-e NODE_NAME=pxc-1 \
-d jonhartley/kubernetes-pxc

docker run --net pxccluster \
--name pxc-2 \
--hostname pxc-2 \
-v /tmp/pxc3:/var/lib/mysql \
-e CONFIG="max_allowed_packet=1G,innodb_buffer_pool_size=500M,max_connections=1000,wsrep_slave_threads=4" \
-e MYSQL_ROOT_PASSWORD=password \
-e SSTUSER_PASSWORD=password \
-e CLUSTER_NAME=pxc_cluster \
-e CPU_LIMIT=2 \
-e MEM_LIMIT=2147483648 \
-e AUTOTUNE=bufferpool \
-e SSSVC=pxc-0 \
-e NODE_NAME=pxc3 \
-d jonhartley/kubernetes-pxc

⁠Restore from Backup on boot

This also inserts a replication user.


docker run \
--name pxc-0 \
--hostname pxc-0 \
-v /var/lib/docker/volumes/pxc-0:/var/lib/mysql \
-v /var/lib/docker/volumes/backup:/var/backup \
-p 33306:3306 \
-e CONFIG="max_allowed_packet=1G,innodb_buffer_pool_size=500M,max_connections=1000,wsrep_slave_threads=4" \
-e MYSQL_ROOT_PASSWORD=password \
-e SSTUSER_PASSWORD=password \
-e CLUSTER_NAME=pxc_cluster \
-e SSSVC=cluster1 \
-e NODE_NAME=pxc-0 \
-e REPLUSER="replication:password" \
-e STARTFROMBACKUP="true" \
-e BACKUPDIR="/var/backup" \
-e BINLOG="true" \
-d jonhartley/kubernetes-pxc

This cluster will expand as required as above by adding CLUSTER_JOIN="hostname or IP of first node"
Once up you can use the script /xtrabackup.sh to backup the DB to the BACKUPDIR
We can share this volume (or in Kubenetes an NFS PV) to another container and make another cluster a slave of the first


docker run \
--name slave-0 \
--hostname slave-0 \
--link pxc-0 \
-v /var/lib/docker/volumes/slave-0:/var/lib/mysql \
-v /var/lib/docker/volumes/backup:/var/backup \
-p 33307:3306 \
-e CONFIG="max_allowed_packet=1G,innodb_buffer_pool_size=500M,max_connections=1000,wsrep_slave_threads=4" \
-e MYSQL_ROOT_PASSWORD=password \
-e SSTUSER_PASSWORD=password \
-e CLUSTER_NAME=pxc_cluster2 \
-e SSSVC=cluster2
-e NODE_NAME=slave-0 \
-e SLAVE="true" \
-e REPLUSER="replication:password" \
-e MASTER_HOST="pxc-0" \
-e STARTFROMBACKUP="true" \
-e BACKUPDIR="/var/backup" \
-e BINLOG="true" \
-d jonhartley/kubernetes-pxc

This second cluster can be expanded as before

⁠Kubernetes

SSL - Various ways to get a cert/key + ca into a container heres one I got to work
I used OpenSSL to create the csr

#>openssl req -new -newkey rsa:2048 -nodes -sha256 \
-keyout cluster.key \
-out cluster.csr \
-subj "/C=GB/ST=Some County/L=Town/street=Street/O=Company/OU=IT/CN=cluster.data.cluster.local/subjectAltName=DNS.1=pxc-0.cluster.data.cluster.local,DNS.2=pxc-1.cluster.data.cluster.local,DNS.3=pxc-2.cluster.data.cluster.local"

Use the service name and each node name so it can be used for cross node (WSREP) SSL as well
Ensure key is compatible

#>openssl rsa -in cluster.key -out cluster.key

Then sign using Kubernetes CA (this was done on master node)

#>cat > v3.ext << EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
EOF

#>openssl x509 -req -days 1825 -in cluster.csr -extfile v3.ext -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out cluster.crt -sha256
#>cat cluster.crt cluster.key > cluster pem

Once we have a pem format crt/key combination we add it to Kubernetes as a secret (in the correct namespace)

kubectl -n data create secret generic cluster-ssl --from-file=cluster.pem

Add volume to container to access pem
All PODs have access to kubernetes ca.crt (/run/secrets/kubernetes.io/serviceaccount/ca.crt)

        volumeMounts:
         - name: cluster-ssl
           mountPath: /etc/mysql/cluster.pem
           subPath: cluster.pem
      volumes:
      - name: cluster-ssl
        secret:
          secretName: cluster-ssl
          items:
          - key: cluster.pem
            path: cluster.pem

Below is setup for openstack with cinder PVs, small configuration, it can be scaled easily, always go with odd numbers for quoram, includes telegraf container for monitoring and full SSL + backup volume on NFS


apiVersion: v1
kind: Namespace
metadata:
  name: data
---
apiVersion: v1
kind: Secret
metadata:
  name: pxc-root-creds
  namespace: data
type: Opaque
data:
  password: cGFzc3dvcmQ=
---
apiVersion: v1
kind: Secret
metadata:
  name: pxc-ssd-creds
  namespace: data
type: Opaque
data:
  password: cGFzc3dvcmQ=
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pxcnfs
  namespace: data
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.3.250
    path: "/volume1/kubernetes"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pxcnfs
  namespace: data
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
     storage: 10Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-telegraf
  namespace: data
  labels:
    name: mysql-telegraf
data:
  telegraf.conf: |-
    [global_tags]

    [agent]
        interval = "10s"
        round_interval = true
        metric_batch_size = 1000
        metric_buffer_limit = 10000
        collection_jitter = "0s"
        flush_interval = "10s"
        flush_jitter = "0s"
        precision = ""
        debug = false
        quiet = false
        logfile = ""
        hostname = ""
        omit_hostname = false
    [[inputs.disk]]
      mount_points = ["/var/lib/mysql","/var/backup"]
    [[inputs.mysql]]
      servers = ["monitor:9087345jhveoihj34095tu5@tcp(127.0.0.1:3306)/"]
      gather_user_statistics              = false
      gather_table_schema                 = false
      gather_info_schema_auto_inc         = false
      gather_innodb_metrics               = true
      gather_process_list                 = true
      gather_slave_status                 = false
      gather_binary_logs                  = false
      gather_table_io_waits               = true
      gather_table_lock_waits             = true
      gather_index_io_waits               = false
      gather_event_waits                  = false
      gather_file_events_stats            = false
      gather_perf_events_statements       = false
      interval_slow                       = "30m"
    [[outputs.prometheus_client]]
        listen = ":9126"
---
apiVersion: v1
kind: Service
metadata:
  name: cluster
  namespace: data
  labels:
    app: pxc
spec:
  ports:
    - port: 3306
      name: mysql
    - port: 4444
      name: state-snapshot-transfer
    - port: 4567
      name: replication-traffic
    - port: 4568
      name: incremental-state-transfer
  clusterIP: "None"
  selector:
    app: pxc
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: pxc
  namespace: data
spec:
  serviceName: cluster
  replicas: 3
  selector:
    matchLabels:
      app: pxc
  template:
    metadata:
      annotations:
        prometheus.io/port: "9126"
        prometheus.io/scrape: "true"
      labels:
        app: pxc
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values: 
                    - pxc
              topologyKey: "kubernetes.io/hostname"
      containers:
      - name: pxcnode
        image: jonhartley/kubernetes-pxc
        resources:
          limits:
            memory: "3000Mi"
            cpu: "1"
          requests:
            memory: "3000Mi"
            cpu: "1"
        imagePullPolicy: Always
        volumeMounts:
         - name: datadir
           mountPath: "/var/lib/mysql"
         - name: pxcnfs
           mountPath: "/var/backup"
           subPath: pxcbackup
         - name: cluster-ssl
           mountPath: /etc/mysql/cluster.pem
           subPath: cluster.pem
           readOnly: true
        ports:
          - containerPort: 3306
          - containerPort: 4444
          - containerPort: 4567
          - containerPort: 4568
        livenessProbe:
          exec:
            command:
            - /usr/bin/livecheck.sh
          initialDelaySeconds: 60
          timeoutSeconds: 5
          periodSeconds: 10
        readinessProbe:
          exec:
            command:
            - /usr/bin/clustercheck.sh
          initialDelaySeconds: 120
          timeoutSeconds: 5
          periodSeconds: 10
        env:
          - name: CONFIG
            value: "max_allowed_packet=1G,innodb_buffer_pool_size=512M,max_connections=500,innodb_log_file_size=256M,ssl-ca=/run/secrets/kubernetes.io/serviceaccount/ca.crt,ssl-cert=/etc/mysql/cluster.pem,ssl-key=/etc/mysql/cluster.pem"
          - name: WSREPCONFIG
            value: "wsrep_slave_threads=2"
          - name: CLUSTER_NAME
            value: "cluster"
          - name: MYSQL_ROOT_PASSWORD
            valueFrom:
              secretKeyRef:
                name: pxc-root-creds
                key: password
          - name: SSTUSER_PASSWORD
            valueFrom:
              secretKeyRef:
                name: pxc-ssd-creds
                key: password
          - name: MEM_LIMIT
            valueFrom:
              resourceFieldRef:
                containerName: pxcnode
                resource: limits.memory
          - name: MY_CPU_LIMIT
            valueFrom:
              resourceFieldRef:
                containerName: pxcnode
                resource: limits.cpu
          - name: BACKUPDIR
            value: "/var/backup"
          - name: AUTOTUNE
            value: "bufferpool"
          - name: SSL
            value: "all"
      - name: telegraf
        image: telegraf:1.7.3-alpine
        resources:
          limits:
            memory: "20Mi"
            cpu: "20m"
          requests:
            memory: "20Mi"
            cpu: "20m"
        imagePullPolicy: Always
        volumeMounts:
         - name: config
           mountPath: /etc/telegraf
         - name: datadir
           mountPath: "/var/lib/mysql"
         - name: pxcnfs
           mountPath: "/var/backup"
           subPath: pxcbackup
        ports:
          - containerPort: 9126
      volumes:
      - name: config
        configMap:
          name: mysql-telegraf
      - name: pxcnfs
        persistentVolumeClaim:
          claimName: pxcnfs
      - name: cluster-ssl
        secret:
          secretName: cluster-ssl
          items:
          - key: cluster.pem
            path: cluster.pem
  volumeClaimTemplates:
  - metadata:
      name: datadir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "cinder"
      resources:
        requests:
          storage: 10Gi

Tag summary

Recent tags

Content type

Image

Digest

Size

252.8 MB

Last updated

about 7 years ago

docker pull jonhartley/kubernetes-pxc