Public Repository

Last pushed: 7 months ago
Short Description
Compare MySQL or PostgreSQL databases
Full Description

compare-mysql

IronSource compare-mysql container provides an easy way to compare data between 2 MySQL or 2 PostgreSQL server instances.
Results are an archive with xls spreadsheets for each database comparison information.
Result archive is uploaded to Amazon S3.
Results of comparison could be obtained by email, by direct access to output folder and direct access to S3.

AWS S3:

For development use credentials for Amazon s3 as environment variables

-e AWS_ACCESS_KEY_ID= -e AWS_SECRET_ACCESS_KEY= -e AWS_DEFAULT_REGION=

You can use predefined s3 configurable values or set them as environment variables:

-e AWS_DEFAULT_REGION=us-west-2
-e AWS_BUCKET=icweb-dwh-dashboard
-e AWS_BUCKET_KEY=staging/compare
# uploaded file will be available for this time period
-e AWS_FILE_PERIOD='5 days'
$ docker login 
$ docker pull ironsource/compare-mysql  
$ docker run --name compare-mysql -it -v /[YourPath]/config.yml:/opt/compare-mysql/config/config.yml -v [YourPath]/output:/opt/compare-mysql/output -v [YourPath]/logs:/opt/compare-mysql/logs ironsource/compare-mysql

Configuration:

  • Make sure to have the config.yml in the path mentioned in the docker run -v flag.
  • The output will be written into the path mentioned in second docker run -v flag.

In config.yml:

  • Fill in databases, tables and columns names:
    • Group columns to group by and measure columns to compare by.
  • servers: connection configuration for the MySQL or PostgreSQL servers.
  • email_config: fill in only if you want to receive an email with the comparison results.
  • compareDate: enables to compare by a specific generic time, for example "yesterday"
    • Result of comparison will be in output/specific-date directory
    • In case if you use only filters - result will be in output/latestCompareResult directory
# list of available schemas in case of postgres compare
# list of available databases in case of mysql compare
databases:
    [databaseName1]:
        [TableName1]:
            Group:
                # - The same column in both servers
                - [ColumnName1]
                # Column in previous server: Column in current server
                - [ColumnName2]: [ColumnName2]
            Measures:
                - [ColumnName3]
                - [ColumnName4]
            Filters:
                # - The same filter for both servers
                - [ColumnName5] != [SomeValue]
                #
                # OR per server
                #
                # ServerName: Filter for this server
                [ServerName1]: [ColumnName5] = [SomeValue]
        [TableName2]:
            Group:
                - [ColumnName1]
                - [ColumnName2]
            Measures:
                - [ColumnName3]
                - [ColumnName4]
            Filters:
                - [ColumnName5] != [SomeValue]
    [databaseName2]:
        [TableName1]:
            Group:
                - [ColumnName1]
                - [ColumnName2]
            Measures:
                - [ColumnName3]
                - [ColumnName4]
            Filters:
                - [ColumnName5] != [SomeValue]

# list of available servers
servers:
    [ServerName1]:
        host: 
        port: 3306
        user: 
        pass:
        # type = mysql || pgsql
        type: mysql
        # if type = pgsql you should add database
        #db: ic
        # in case comparing different schemas in postgres or different databases in mysql
        #schema: ic
    [ServerName2]:
        host: 
        port: 3306
        user: 
        pass:
        type: mysql
        #db: ic
        #schema: ic

email_config:
    host: mail.installcore.com
    port: 25
    from: qa-compare@ironsrc.com
    subject: Compare: Old ETL vs New ETL v1.0


toCompare:
    servers:
        # first server is previous
        - [ServerName1]
        # second server is current
        - [ServerName2]
    databases:
        [databaseName1]:
            - [TableName1]
            - [TableName2]
        [databaseName2]:
            - [TableName1]
emails:
    - jhon@ironsrc.com
    - ted@ironsrc.com
compareDate:
    # uncomment in order to compare by yesterday's date
    # key is the column name with date
    Date: 'yesterday'
Docker Pull Command
Owner
ironsource