astuetz/data-lake-cache repository overview

⁠A smart cache (ALPHA)

A smart cache for data lakes like Google Big Query. Currently, the data-lake-cache is only a draft. Don't use it in production.

Proxies all requests to Google Big Query and caches the result in a local DB. The data must be partitioned by time/day for this cache to work effectively. The service supports a Rest-API for querying. When a request is sent to this API endpoint, the cache will be checked for the data. If there is no cache entry yet, the query will be sent to the cloud DB without modifications. After the response is returned, the service evaluates if the used partitions should be cached. If so the service will trigger a new query to get all columns for the cache. Requests to cached partitions will be handled by the cache.

⁠Modes

In the future the service will provide different modes or caching strategies to adapt to different needs. The cache is limited to a specific size in bytes. Currently, if a new query exceeds the size it won't be cached.

⁠Fast Queries

If a query isn't cached, it will first send the query to the cloud DB and make an extra request to the cloud db to retrieve all columns for the partitions. This mode has the advantage to keep the latency low for cache misses, but the disadvantage of higher costs.

⁠Low Cost

(Not yet implemented) The query will always be answered from cache, except if the size limit of the cache is exceeded. This means that all columns of the specified partitions need to be cached and then the cache will be queried.

⁠Supported Cache DBs

• Postgres

⁠Supported Cloud DBs

• Google Big Query

Not supported yet:

• Amazon Athena

⁠Current limitations

• Limitations: first version caches all columns
• Limitations: only complete cache hit, no delta calculation implemented
• Limitations: requires date between x and y clause

⁠Setup and Usage

This application is using java with spring boot. It can be started like any other Spring Boot application.

⁠Build

To build and test: ./gradlew clean build a fat jar should be generated here: build/libs/cloud-cache-*.jar

⁠Configuration

Configuration settings can be found in application.properties. Some of the most important properties:

• target.columns The datatypes and columns of the data lake table(s) need to be specified here. The datatypes need to be specified in postgres style e.g. {'event_type':'VARCHAR(30) NOT NULL', 'value':'INT NOT NULL', 'date':'DATE NOT NULL', 'ids':'int[] NOT NULL'}
• target.summableColumns Specify the column names which can be summed up. The other columns will be grouped, hence the result set is smaller in the cache.
• target.partitionColumnName The column name which is partitioned, must be of type date
• spring.cloud.gcp.bigquery.datasetName The dataset name which contains the target tables
• cache.maxSizeMB The maximum size of the cache in MB.
• cache.InvalidAfterHours How long are cached entries valid
• spring.datasource.url Url to the postgres instance which is used for caching
• spring.datasource.username Postgres username
• spring.datasource.password Postgres password
• googleStorage.tmpCacheBucketName Google Bucket name which is used for temporarily storing BQ extracts. It is recommended to set a lifecycle policy for deletion of one day.

Configure cloud access. e.g. for Google Cloud set the environment variable GOOGLE_APPLICATION_CREDENTIALS

If you use the docker image, set the appropriate java options in INSTANCE_JAVA_OPTIONS e.g. -Xmx 8g

⁠Run

You can either build and start the jar file directly with java -jar or use the docker image⁠. Under the directory docker there is a docker-compose file which contains a sample configuration for the postgres db.

⁠Sample Big Query Schema

[
{
    "name": "event_type",
    "type": "STRING",
    "mode": "REQUIRED"
},
{
    "name": "value",
    "type": "INTEGER",
    "mode": "REQUIRED"
},
{
    "name": "date",
    "type": "DATE",
    "mode": "REQUIRED"
}
]

⁠Upload Sample Data

Keep in mind: In the sandbox BQ mode, only dates from 60 days ago till now can be uploaded!

curl -X PUT -H "Content-Type: application/json" http://localhost:8080/data-lake-cache/upload -T sample-data.json

⁠Example queries

curl -X POST -H "Content-Type: application/json" http://localhost:8080/data-lake-cache/proxy -d "{ \"query\": \"select * from cache_test.cache_test_1 where date between '2020-01-01' AND '2020-12-01'\", \"parameters\": {}}"