Sign inSign up

nixyslab/nxs-data-anonymizer

By nixyslab

•Updated about 1 year ago

https://github.com/nixys/nxs-data-anonymizer

Image
0

10K+

nixyslab/nxs-data-anonymizer repository overview

photo_2023-07-28_15-28-52

⁠Quick reference

⁠Supported tags and respective Dockerfile links

⁠What is nxs-data-anonymizer?

nxs-data-anonymizer is a tool to anonymize a PostgreSQL and MySQL/MariaDB/Percona databases dump.

⁠How to use this image

You are able to use Docker image either directly from console or integrate it into your CI/CD.

Also you may use the tool as a binary. See GitHub repo⁠ for details.

Before using you need to inspect your database structure and set up⁠ the nxs-data-anonymizer config in accordance with the sensitive data you need to anonymize.

⁠Console

For example, use the following command if you need to anonymize your PostgreSQL database from production to dev on fly (PostgreSQL Client need to be installed):

docker run -it -v "/path/to/nxs-data-anonymizer.conf:/nxs-data-anonymizer.conf" nixyslab/nxs-data-anonymizer sh -c 'export PGPASSWORD=password; pg_dump -U postgres prod | /nxs-data-anonymizer -t pgsql -c /nxs-data-anonymizer.conf | psql -U postgres dev'

⁠GitLab CI

This section describes how to integrate nxs-data-anonymizer into your GitLab CI. You may add jobs below into your .gitlab-ci.yml and adjust it for yourself.

⁠Job: anonymize prod

Job described in this section do the following:

  • Run when special tag for main branch is set
  • Create a production database dump, anonymize it and updaload into s3 bucket

Job sample:

anonymize:
  stage: anonymize
  image: nixyslab/nxs-data-anonymizer:latest
  variables:
    GIT_STRATEGY: none
    PG_HOST: ${PG_HOST_PROD}
    PG_USER: ${PG_USER_PROD}
    PGPASSWORD: ${PG_PASS_PROD}
  before_script: 
  - echo "${S3CMD_CFG}" > ~/.s3cmd
  - echo "${NXS_DA_CFG}" > /nxs-data-anonymizer.conf
  script:
  - pg_dump -h ${PG_HOST} -U ${PG_USER} --schema=${PG_SCHEMA} ${PG_DATABASE} | /nxs-data-anonymizer -t pgsql -c /nxs-data-anonymizer.conf | gzip | s3cmd put - s3://bucket/anondump.sql.gz
  only:
  - /^v.*$/
  except:
  - branches
  - merge_requests
⁠Job: update stage

Job described in this section do the following:

  • Manual job for stagebranch
  • Download the anonymized dump from s3 bucket and load into stage database

Job sample:

restore-stage:
  stage: restore
  image: nixyslab/nxs-data-anonymizer:latest
  variables:
    GIT_STRATEGY: none
    PG_HOST: ${PG_HOST_STAGE}
    PG_USER: ${PG_USER_STAGE}
    PGPASSWORD: ${PG_PASS_STAGE}
  before_script: 
  - echo "${S3CMD_CFG}" > ~/.s3cmd
  script:
  - s3cmd --no-progress --quiet get s3://bucket/anondump.sql.gz - | gunzip | psql -h ${PG_HOST} -U ${PG_USER} --schema=${PG_SCHEMA} ${PG_DATABASE}
  only:
  - stage
  when: manual
⁠CI/CD variables

This section contains a description for CI/CD variables used in GitLab CI job samples above.

⁠General
VariableDescription
S3CMD_CFGS3 storage config
PG_SCHEMAPgSQL schema
PG_DATABASEPgSQL database name
⁠Production
VariableDescription
NXS_DA_CFGnxs-data-anonymizer config
PG_HOST_PRODPgSQL host
PG_USER_PRODPgSQL user
PG_PASS_PRODPgSQL password
⁠Stage
VariableDescription
PG_HOST_STAGEPgSQL host
PG_USER_STAGEPgSQL user
PG_PASS_STAGEPgSQL password

⁠GitHub Actions

This section describes how to integrate nxs-data-anonymizer into your GitHub Actions. You may add jobs presented below into your .github workflows and adjust it for yourself.

⁠Job: anonymize prod

Job described in this section is able to perform the following tasks:

  • Run when special tag is set
  • Create a production database dump, anonymize and upload it into s3 bucket
on:
  push:
    tags:
    - v*.*

jobs:
  anonymize:
    runs-on: ubuntu-latest
    container:
      image: nixyslab/nxs-data-anonymizer:latest
      env:
        PG_HOST: ${{ secrets.PG_HOST_PROD }}
        PG_USER: ${{ secrets.PG_USER_PROD }}
        PGPASSWORD: ${{ secrets.PG_PASS_PROD }}
        PG_SCHEMA: ${{ secrets.PG_SCHEMA }}
        PG_DATABASE: ${{ secrets.PG_DATABASE }}
    steps:
    - name: Create services configs
      run: |
        echo "${{ secrets.S3CMD_CFG }}" > ~/.s3cmd
        echo "${{ secrets.NXS_DA_CFG }}" > /nxs-data-anonymizer.conf
    - name: Anonymize
      run: |
        pg_dump -h ${PG_HOST} -U ${PG_USER} --schema=${PG_SCHEMA} ${PG_DATABASE} | /nxs-data-anonymizer -t pgsql -c /nxs-data-anonymizer.conf | gzip | s3cmd put - s3://bucket/anondump.sql.gz
⁠Job: update stage

Job described in this section deals with the following:

  • Manual job
  • Download the anonymized dump from s3 bucket and load into stage database
on: workflow_dispatch

jobs:
  restore-stage:
    runs-on: ubuntu-latest
    container:
      image: nixyslab/nxs-data-anonymizer:latest
      env:
        PG_HOST: ${{ secrets.PG_HOST_STAGE }}
        PG_USER: ${{ secrets.PG_USER_STAGE }}
        PGPASSWORD: ${{ secrets.PG_PASS_STAGE }}
        PG_SCHEMA: ${{ secrets.PG_SCHEMA }}
        PG_DATABASE: ${{ secrets.PG_DATABASE }}
    steps:
    - name: Create services configs
      run: |
        echo "${{ secrets.S3CMD_CFG }}" > ~/.s3cmd
    - name: Restore
      run: |
        s3cmd --no-progress --quiet get s3://bucket/anondump.sql.gz - | gunzip | psql -h ${PG_HOST} -U ${PG_USER} --schema=${PG_SCHEMA} ${PG_DATABASE}
⁠GitHub Actions secrets

This section contains a description of secrets used in GitHub Actions job samples above.

⁠General
VariableDescription
S3CMD_CFGS3 storage config
PG_SCHEMAPgSQL schema
PG_DATABASEPgSQL database name
⁠Production
VariableDescription
NXS_DA_CFGnxs-data-anonymizer config
PG_HOST_PRODPgSQL host
PG_USER_PRODPgSQL user
PG_PASS_PRODPgSQL password
⁠Stage
VariableDescription
PG_HOST_STAGEPgSQL host
PG_USER_STAGEPgSQL user
PG_PASS_STAGEPgSQL password

⁠License

nxs-data-anonymizer⁠ is open source and released under the terms of the Apache License 2.0⁠.

As with all Docker images, these likely also contain other software which may be under other licenses (such as Bash, etc from the base distribution, along with any direct or indirect dependencies of the primary software being contained).

As for any pre-built image usage, it is the image user's responsibility to ensure that any use of this image complies with any relevant licenses for all software contained within.

Tag summary

Content type

Image

Digest

sha256:e2e59886d…

Size

36.9 MB

Last updated

about 1 year ago

docker pull nixyslab/nxs-data-anonymizer