datopian/bitstore repository overview

BitStore is a DataHub microservice for storing blobs i.e. files. It is a lightweight auth wrapper for an S3-compatible object store that integrates with the rest of the DataHub stack and especially the auth service.

⁠Quick Start

⁠Clone the repo and install

make install

⁠Run tests

make test

⁠Run server

python server.py

⁠Env Vars

• AUTH_SERVER - the FQ URL of the auth server. Used for looking up the public key for communicating with the auth server from the auth server.
• Object store: connection info for the underlying S3-style objectstore service

  STORAGE_ACCESS_KEY_ID
  STORAGE_SECRET_ACCESS_KEY
  STORAGE_BUCKET_NAME
  

  Copy
• STORAGE_PATH_PATTERN - pattern for generating the storage path in the objectstore for a given rile. That is, object_store_path = make_path(STORAGE_PATH_PATTERN.format{fileinfo}). May contain any format string available for a file in authorize API including
  • {path} (relative path to file in package)
  • {md5}.
  • {basename} which is the filename, extracted from the {path}
  • {dirname} which is the dirname, extracted from the {path}
  • {extension} which is the extension of the filename
  • {md5} (and {md5_hex} which is the md5 in hex form)
    Note: in addition to file info the owner and dataset (name) are available as {owner} and {dataset}. Examples:
  • custom/path/{owner}/{dataset}/{path} will, given {owner: datahq, name: datax, path: data/file.csv} will end up with custom/path/datahq/datax/data/file.csv
  • {md5} - storage path is md5 hash of the file (assuming md5 hash is provided)

Note: requested permissions to auth server will be like:

permissions:
  datapackage-upload
service:
  SERVICE_NAME (config defined above e.g. 'rawstore')

⁠API

⁠Get authorized upload URL(s)

/authorize

Method: POST

Query Parameters:

• jwt - permission token (received from /user/authorize)

Headers:

• Auth-Token - permission token (can be used instead of the jwt query parameter)

Body:

JSON content with the following structure:

{
    "metadata": {
        "owner": "<user-id-of-uploader>",
        "name": "<data-set-unique-id>"
    },
    "filedata": {
        "<relative-path-to-file-in-package-1>": {
            "length": 1234, #length in bytes of data
            "md5": "<md5-hash-of-the-data>",
            "type": "<content-type-of-the-data>",
            "name": "<file-name>"
        },
        "<relative-path-to-file-in-package-2>": {
            "length": 4321,
            "md5": "<md5-hash-of-the-data>",
            "type": "<content-type-of-the-data>",
            "name": "<file-name>"
        }
        ...
    }
}

owner must match the userid that is in the authentication token.

Returns

Signed urls to upload into S3:

{
  fileData: {
    "<file-name-1>": {
      "md5-hash": "...",
      "name": "<file-name>",
      "type": "<file-type>",
      "upload_query": {
        'Content-MD5': '...',
        'Content-Type': '...',
        'acl': 'public-read',
        'key': '<path>',
        'policy': '...',
        'x-amz-algorithm': 'AWS4-HMAC-SHA256',
        'x-amz-credential': '...',
        'x-amz-date': '<date-time-in-ISO',
        'x-amz-signature': '...'
      },
      "upload_url": "<s3-url>",
      "exists": true/false
    },
    "<file-name-2>": ...,
    ...
  }
}

⁠Get information regarding the datastore

/info

Method: GET

Query Parameters:

• jwt - permission token (received from /user/authorize)

Headers:

• Auth-Token - permission token (can be used instead of the jwt query parameter)

Returns:

JSON content with the following structure:

{
    "prefixes": [
        "https://datastore.openspending.org/123456789",
    ]
}

prefixes is the list of possible prefixes for an uploaded file for this user.

⁠Check and Generate S3 Presigned URL for private objects

/presign

Methos: GET

Query Parameters:

• jwt - permission token (received from /user/authorize)
• url - original URL for S3 object
• ownerid - authenticated user Id

Headers:

• Auth-Token - permission token (can be used instead of the jwt query parameter)

Returns:

Original or Pre-Signed S3 URL:

{
    "url": "https://s3.amazonaws.com/rawstore/ownername/dataset/maydata.csv?x=y",
}