liveseo/parquet-api repository overview

⁠Parquet-api

Parquet-api is a simple library compatible with AWS S3 that uses DuckDB and Node.js to build a docker api that you can make sql queries using http requests.

You can create local tables and connect to remote parquet files as DuckDB⁠ allow you.

Local parquet files need to be read/write without concurrency, so if you use LOCAL_DB as 1 I don't recommend to use this script in production because requests can be slow if is not in an fast ssd. Memory DB (LOCAL_DB as 0) can be up to 50% faster.

Make sure to not open 8080 port as public without SSL.

To run, ou can execute:

docker run -d -p 0.0.0.0:8080:8080 liveseo/parquet-api

You can also add AWS S3 keys to make it connected with keys S3_REGION, S3_KEY and S3_SECRET:

docker run -d -p 0.0.0.0:8080:8080 -e S3_SECRET=mysecret123 -e S3_KEY=mykey -e S3_REGION=sa-east-1 liveseo/parquet-api

If you want to use a simple authorization token, set TOKEN env and set it as header on every request:

docker run -d -p 0.0.0.0:8080:8080 -e TOKEN=mysecrettoken123 liveseo/parquet-api

If you want to use memory database with more workers for fast response set LOCAL_DB env as 0 and set WORKERS to any number higher than 1:

docker run -d -p 0.0.0.0:8080:8080 -e WORKERS=10 -e LOCAL_DB=0 liveseo/parquet-api

Now, you can make requests in http://localhost:8080/?q=SELECT 2 as abc The response is JSON, like:

[ { "abc": 2 } ]

Source code⁠

Author: liveSEO⁠