datopian/data-api repository overview

⁠Data API

For data management systems including CKAN⁠.

⁠Features

• GraphQL endpoint
• Bulk export of data to json/csv/xlsx files
• datastore_search endpoint (similar to CKAN Datastore extention⁠)

⁠Version

The current version is v1

APP_VERSION = 'v1'

⁠Usage

⁠Endpoints

• /{APP_VERSION}/graphql
• /{APP_VERSION}/download
• /{APP_VERSION}/datastore_search
• /{APP_VERSION}/datastore_search/help

⁠GraphQL Endpoint

GraphQL Endpoint exposes the hasura GraphQL API.

For the GraphQL documentation please refer to Hasura Documentation⁠

⁠datastore_search endpoint

⁠Parameters

• resource_id (string) – MANDATORY id or alias of the resource to be searched against
• q (string or dictionary) – JSON format query restrictions {“key1”: “a”, “key2”: “b”}, it’ll search on each specific field (optional)
• distinct_on [bool || list of field names] – If True: return only distinct rows, if list of fields will return only
• limit (int) – Maximum number of rows to return (optional, default: 100)
• offset (int) – offset this number of rows (optional)
• fields (list of strings) – fields to return (optional, default: all fields)
• sort (string) – comma separated field names with ordering e.g.: “fieldname1, fieldname2 desc” Not implemented yet
• filters (dictionary) – matching conditions to select, e.g {“key1”: “a”, “key2”: “b”} (optional) Not implemented - similar to q

⁠Results:

The result is a JSON document containing:

• schema (JSON) – The data schema
• data (JSON) – matching results in JSON format
• fields (list of dictionaries) – fields/columns and their extra metadata Not Implemented
• offset (int) – query offset value Not implemented
• total (int) – number of total matching records Not implemented

⁠Examples

⁠Help page

⁠Basic query with limit

With a test table having the following schema:

We can make different queries:

⁠Query Table

⁠Query Table with Limit

⁠Query Table with Limit and Offset (pagination)

⁠Contributing

1. copy env variables - cp .env.example .env
2. set up background services - this repo contains a mock environment you can launch with bash run-mock-environment.sh or if connecting to an existing one edit the URLs in the .env file
3. install dependencies - yarn
4. yarn start to launch server / yarn test to run tests
5. setup automatic code formatter - install and use prettier⁠, if using vscode install prettier-vscode extention⁠

Please don't forget to add new variables to .env.example - they also will be used in ci tests.

⁠GitHub CI

⁠Tests

Tests and formating check and run automatically on every push and pull request to master. They run on Docker hub. See documentation here https://docs.docker.com/docker-hub/builds/automated-testing/⁠

To simulate tests as running on dockerhub you can run bash run-tests-in-docker.sh file.

If a pull request has failed checks it shows an error message in GitHub. The link to DockerHub does not work though. You will need to navigate there:

• Docker repository is here https://hub.docker.com/repository/docker/datopian/data-api⁠
• To see build jobs go to builds https://hub.docker.com/repository/docker/datopian/data-api/builds⁠ and find your build/test

⁠Docker image builds

After every push to master⁠ or a pull request to this branch and successful tests there is a new docker image built here https://hub.docker.com/repository/docker/datopian/data-api/builds⁠

⁠Building

New images can be build with the provided Dockerfile⁠ or fetched from dockerhub⁠.

⁠Deployment

Can be deployed as a usual docker container. An environement for this microservice should contain:

1. postgres database
2. hasura
3. environment variables (see .env.example⁠)

⁠License

This project is licensed under the MIT License - see the LICENSE⁠ file for details