lrisdata/exporter repository overview

⁠Liferay Data Warehouse Data Exporter

This python application exports tables from a relational database to the Liferay Data Warehouse.

⁠Configuration through DXP Cloud Secrets

There are 3 main sections that must be configured: DATABASE, ETL, and GCP. The DATABASE section is specific to the database that is being exported and must be configured by you.

The ETL and GCP configuration information must be assigned to you. Contact the data operations team to get the configuration options specific to your system(s).

All configuration variables are required except EXPORTER_DATABASE_PASSWORD and EXPORTER_DATABASE_TABLES.

If the database does not have a password (not recommended for production!), do not set EXPORTER_DATABASE_PASSWORD.

To export all tables in the database, do not set EXPORTER_DATABASE_TABLES. To export a subset of tables in the database, list the desired table names separated by commas in EXPORTER_DATABASE_TABLES.

⁠Sample Environment Variable Configuration

EXPORTER_LOG_DIRECTORY=...

EXPORTER_SYNC_TIME=21:00

EXPORTER_DATA_SOURCE_TYPE=database

EXPORTER_DATABASE_NAME=my_database
EXPORTER_DATABASE_HOST=localhost
EXPORTER_DATABASE_PORT=port
EXPORTER_DATABASE_USER=user
EXPORTER_DATABASE_PASSWORD=password
EXPORTER_DATABASE_TABLES=table1, table2

EXPORTER_ETL_GROUP=...
EXPORTER_ETL_PROJECT=...
EXPORTER_ETL_ENVIRONMENT=...
EXPORTER_ETL_VERSION=...
EXPORTER_ETL_JOB=...

EXPORTER_GCP_DATASET=...
EXPORTER_GCP_PROJECT_ID=...
EXPORTER_GCP_PUBLISHER_PROJECT_ID=...
EXPORTER_GCP_PUBLISHER_TOPIC=...
EXPORTER_GCP_STORAGE_BUCKET=...
EXPORTER_GCP_SERVICE_ACCOUNT_INFO={...}