ingomuellernet/presto repository overview

⁠Simple Dockerized Presto

This projects aims to make it easy to get started with Presto⁠. It is based on Docker and Docker compose⁠. Currently, the following features are supported:

• Dedicated Presto scheduler node and variable number of worker nodes
• Function Namespace Manager⁠ (for creating functions⁠)
• Hive connector⁠, Hive Metastore, and pseudo-replicated HDFS (i.e., without replication) with variable number of data nodes
• Reading from S3 without addtitional configuration (if running in EC2 and with a properly configured instance profile⁠)

⁠Starting Presto

The following should be enough to bring up all required services:

docker-compose up

⁠Varying the Number of Workers and Data Nodes

To change the number of Presto worker nodes or HDFS data nodes, use the --scale flag of docker-compose:

docker-compose up --scale datanode=3 --scale presto-worker=3

⁠Building the Image Locally

Above command uses a pre-built docker image⁠. If you want the image to be build locally, do the following instead:

docker-compose --file docker-compose-local.yml up

If you are behind a corporate firewall, you will have to configure Maven (which is used to build part of Presto) as follows before running above command:

export MAVEN_OPTS="-Dhttp.proxyHost=your.proxy.com -Dhttp.proxyPort=3128 -Dhttps.proxyHost=your.proxy.com -Dhttps.proxyPort=3128"

⁠Uploading Data to HDFS

The data/ folder is mounted into the HDFS namenode container, from where you can upload it using the HDFS client in that container (docker-presto_presto_1 may have a different name on your machine; run docker ps to find out):

docker exec -it docker-presto_namenode_1 hadoop fs -mkdir /dataset
docker exec -it docker-presto_namenode_1 hadoop fs -put /data/file.parquet /dataset/
docker exec -it docker-presto_namenode_1 hadoop fs -ls /dataset

⁠Running Queries

You can use the Presto CLI included in the Docker containers of this project (adapt container name if necessary):

docker exec -it docker-presto_presto_1 presto-cli --catalog hive --schema default

Alternatively, you can download the Presto CLI⁠, rename it, make it executable, and run the following:

./presto-cli --server localhost:8080 --catalog hive --schema default

⁠Creating an External Table

Suppose you have the following file test.json:

{"s": "hello world", "i": 42}

Upload it to /test/test.csv on HDFS as described above. Then run the following in the Presto CLI:

CREATE TABLE test (s VARCHAR, i INTEGER) WITH (EXTERNAL_LOCATION = 'hdfs://namenode/test/', FORMAT = 'JSON');

For external tables from S3, spin up this service in an EC2 instance, set up an instance profile⁠ for that instance, and use the s3a:// protocol instead of hdfs://.

⁠Adminstrating the MySQL Databases

In case you need to make manual changes or want to inspect the MySQL databases, you can connect to it like this:

docker exec -it docker-presto_mysql_1 mysql -ppassword