dimitri/pgloader

By dimitri

Updated over 2 years ago

Automatic build of current pgloader master's tree available as a docker image.

Image
23

100K+

PGLoader

pgloader is a data loading tool for PostgreSQL, using the COPY command.

Its main advantage over just using COPY or \copy, and over using a Foreign Data Wrapper, is its transaction behaviour, where pgloader will keep a separate file of rejected data, but continue trying to copy good data in your database.

The default PostgreSQL behaviour is transactional, which means that any erroneous line in the input data (file or remote database) will stop the entire bulk load for the table.

pgloader also implements data reformatting, a typical example of that being the transformation of MySQL datestamps 0000-00-00 and 0000-00-00 00:00:00 to PostgreSQL NULL value (because our calendar never had a year zero).

LICENCE

pgloader is available under The PostgreSQL Licence.

Usage

You can either give a command file to pgloader or run it all from the command line, see the pgloader quick start on https://pgloader.readthedocs.io for more details.

$ docker run --rm -it dimitri/pgloader:latest pgloader --help
$ docker run --rm -it dimitri/pgloader:latest pgloader --version

For example for a full migration from MySQL, including schema definition (tables, indexes, foreign keys, comments) and parallel loading of the corrected data:

$ docker run --rm -it dimitri/pgloader:latest \
     pgloader \
       mysql://root:$MYSQL_ROOT_PASSWORD@172.17.0.2/dbname \
       pgsql://postgres:$POSTGRES_PASSWORD@172.17.0.3/dbname

In some cases using a version of pgloader built with Clozure-CL instead of SBCL can be worthwile, so it's available too:

$ docker run --rm -it dimitri/pgloader:ccl.latest pgloader --version

Docker Pull Command

docker pull dimitri/pgloader