datopian/metastore repository overview

⁠DataHub metastore

A search services for DataHub.

Searches Elasticsearch and returns matching documents (returned document content structure are not defined by this module)

⁠Quick Start

⁠Clone the repo and install

make install

⁠Run tests

make test

⁠Run server

python server.py

⁠API

Elasticsearch: version 5.x should be installed

Endpoint: /metastore/search

Method: GET

HEADER: Auth-Token (received from /auth/check)

Query Parameters:

• q - match-all query string Will search the following properties:

  • title
  • datahub.owner
  • datahub.ownerid
  • datapackage.readme

• size - number of results to return [max 100]

• from - offset to start returning results from

all other parameters will be treated as filters for the query (requiring exact match of value)

Returns: All packages that match the filter:

{
  "summary": {
    "total": "total-number-of-matched-documents",
    "totalBytes": "total-size-of-matched-datasets"
  },
  "results": [
    "list of matched documents"
  ]
}

Endpoint: /metastore/search/events

Method: GET

HEADER: Auth-Token (received from /auth/check)

Query Parameters:

• q - match-all query string
• event_entity - flow|account|etc... (currently only flow is supported)
• event_action - create|finished|deleted|etc... (currently only finished is supported)
• owner - ownerid (usually hash of user's Email)
• dataset - dataset name
• status - OK|Not OK
• findability - published|unlisted|private

Query Parameters for pagination and sorting:

• sort - desc|asc (defaults to desc)
• size - number of results to return [max 100]
• from - offset to start returning results from

Returns: All packages that match the filter:

{
  "results": [
    {
      "dataset": "finance-vix",
      "event_action": "finished",
      "event_entity": "flow",
      "findability": "published",
      "messsage": "",
      "owner": "core",
      "ownerid": "core",
      "status": "OK",
      "timestamp": "2017-01-01T00:00:00.000000",
      "payload": {
        "flow-id": "core/finance-vix"
      }
    }
  ],
  "summary": {
    "total": 1,
    "totalBytes": 0
  }
}