Sign inSign up

datopian/specstore

By datopian

•Updated almost 8 years ago

Source Spec API

Image
0

4.2K

datopian/specstore repository overview

⁠DataHQ Spec Store

Build Status

An API server for managing a Source Spec Registry

⁠Quick start

⁠Clone the repo and install

make install

⁠Run tests

make test

⁠Run server

python server.py

⁠Env Vars

  • DATABASE_URL: A SQLAlchemy compatible database connection string (where registry is stored)
  • AUTH_SERVER: The domain name for the authentication server
  • DPP_URL: URL for the datapackage pipelines service (e.g. http://host:post/)

⁠API

⁠Status

/source/{owner}/{dataset-id}/{revision-number}

Note: Also, you can get info about latest and latest successful revisions by hitting following endpoints

  • latest - /source/{owner}/{dataset-id}/latest
  • successful - /source/{owner}/{dataset-id}/successful
⁠Method

GET

⁠Response
{
  "id": "<revision-id>",
  "spec_contents": <source-specifications>,
  "modified": <last-modified>,
  "state": <QUEUED|INPROGRESS|SUCCEEDED|FAILED>,
  "logs": <full-logs>,
  "error_log": [ <error-log-lines> ],
  "stats": {
    "bytes": <number>,
    "count_of_rows": <number>,
    "dataset_name": <string>,
    "hash": <datapackage-hash>
  },
  "pipelines": {
    "<pipeline-id-1>": {
      "title": "Creating CSV",
      "status": "SUCCEEDED",
      "stats": null,
      "error_log": []
    },
    "<pipeline-id-2>": {
      "title": "Creating JSON",
      "status": "INPROGRESS",
      "stats": {},
      "error_log": []
    },
    "<pipeline-id-3>": {
      "title": "Creating ZIP",
      "status": "FAILED",
      "stats": {},
      "error_log": [
        'error',
        'logs'
      ]
    }
  }
}

state definition:

  • QUEUED: Flow created but not running
  • INPROGRESS: Flow is running
  • SUCCEEDED: Finished successfully
  • FAILED: Failed to run
⁠Upload

/source/upload

⁠Method

POST

⁠Headers
  • Auth-Token - permission token (received from conductor)
  • Content-type - application/json
⁠Body

A valid spec in JSON form. You can find example Flow-Spec in README of planer API⁠

⁠Response
{
  "success": true,
  "dataset_id": "<dataset-identifier>",
  "flow_id": "<dataset-identifier-with-revision-number>",
  "errors": [
      "<error-message>"
  ]
}
⁠Update

/source/update

⁠Method

POST

⁠Body

Payload in JSON form.

{
  "pipeline": "<pipeline-id>",
  "event": "queue/start/progress/finish",
  "success": true/false (when applicable),
  "errors": [list-of-errors, when applicable]
}
⁠Response
{
  "success": success/pending/fail,
  "id": "<identifier>"
  "errors": [
      "<error-message>"
  ]
}

Tag summary

Content type

Image

Digest

Size

178.1 MB

Last updated

almost 8 years ago

docker pull datopian/specstore