Sign inSign up

rayyildiz/pgexport

By rayyildiz

•Updated about 6 years ago

Spark 3.0 DEMO to export data from PostgreSQL to Blob Storage or Folder

Image
1

291

rayyildiz/pgexport repository overview

⁠Spark Job for exporting data from PostgreSQL to Azure Blob Storage or folder.

It uses delta lake⁠ to store data.

  • AZ_ACCOUNT_KEY = Azure Account key
  • AZ_ACCOUNT_NAME = Azure Storage account name
  • EXPORT_PATH Export path. Use this pattern to use Azure Blob Storage: wasbs://[email protected]/PATH. CONTAINER is container name⁠ and ACCOUNT_NAME is Azure storage Account Name⁠. Or you can use a physical path for EXPORT_PATH (example: /mnt/apth/to/data)
  • PG_DATABASE, PG_PORT, PG_USERNAME , PG_PASSWORD and PG_DBNAME for PostgreSQL information.

If EXPORT_PATH is not Azure endoint, you don't need to define AZ_ACCOUNT_NAME and AZ_ACCOUNT_KEY environment variable.

First install Spark K8s Operator⁠ and then run kubectl apply -f spark-application.yaml command.

⁠spark-application.yaml
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: postgresql-export
spec:
  type: Scala
  mode: cluster
  image: rayyildiz/pgexport
  imagePullPolicy: Always
  mainClass: dev.rayyildiz.pgexp.ExportData
  mainApplicationFile: "local:///opt/spark/work-dir/pg-exporter.jar"
  sparkVersion: "3.0.0"
  restartPolicy:
    type: Never
  volumes:
    - name: "tmp-volume"
      hostPath:
        path: "/tmp"
        type: Directory
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 3.0.0
    serviceAccount: spark
    volumeMounts:
      - name: "tmp-volume"
        mountPath: "/tmp"
    envVars:
      PG_DATABASE: "IPAddress or domain"
      PG_PORT: "5432"
      PG_USERNAME: "postgres"
      PG_PASSWORD: "postgres"
      PG_DBNAME: "postgres"
      EXPORT_PATH: "wasbs://CONTAINER@ACCOUNT_NAME.blob.core.windows.net/PATH"
      AZ_ACCOUNT_NAME: ACCOUNT_NAME
      AZ_ACCOUNT_KEY: ACCOUNT_KEY
  executor:
    cores: 1
    instances: 3
    memory: "512m"
    labels:
      version: 3.0.0
    volumeMounts:
      - name: "tmp-volume"
        mountPath: "/tmp"
    envVars:
      PG_DATABASE: "IPAddress or domain"
      PG_PORT: "5432"
      PG_USERNAME: "postgres"
      PG_PASSWORD: "postgres"
      PG_DBNAME: "postgres"
      EXPORT_PATH: "wasbs://CONTAINER@ACCOUNT_NAME.blob.core.windows.net/PATH"
      AZ_ACCOUNT_NAME: ACCOUNT_NAME
      AZ_ACCOUNT_KEY: ACCOUNT_KEY

Tag summary

Content type

Image

Digest

Size

330.4 MB

Last updated

about 6 years ago

docker pull rayyildiz/pgexport