bwsw/cs-kv-storage repository overview

⁠Key-value storage

This project provides key-value storages as an extension for Apache CloudStack.

Following storage types are supported:

• ACCOUNT

Persistent storages for Apache CloudStack accounts managed via Apache CloudStack API. Each account can have many storages. This storage type can be configured to save a history of operations.

• VM

Storages for Apache CloudStack virtual machines created and deleted automatically while creating or expunging virtual machines.

• TEMP

Temporal storages with specified TTL created via Apache CloudStack API. TTL can be updated after creation. This operation as well as storage deletion can be done via Apache CloudStack API and key-value storage API.

• Deployment⁠
• Configuration⁠
• Build & Run⁠
• API⁠

⁠Deployment

Following components should be deployed:

• Elasticsearch 6.2

The official documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/6.2/index.html⁠

Once it is deployed next step is to create necessary indexes and templates. To achieve this [initialization script] (/elasticsearch/init.sh) should be executed (use -h option to print how to use it).

• cs-kv-storage

See configuration⁠ and build & run⁠ sections.

⁠Configuration

The example of the configuration file can be found here⁠.

⁠Build & Run

⁠Application as jar file

$ sbt assembly
$ java -Dconfig.file=<config.path> -jar target/scala-2.12/cs-kv-storage-<version>-jar-with-dependencies.jar

where <config.path> and <version> should be replaced with actual values.

⁠Application as docker container

This project provides two options to build docker images:

using a default version

$ docker build -t <tag> .
$ docker -p <port>:8080 -v <config.path>:/opt/cs-kv-storage/application.conf <tag>

where <tag>, <port> and <config.path> should be replaced with actual values

using a ready jar

$ docker build -t <tag> --build-arg APP_PATH=<path to jar with dependencies> .
$ docker -p <port>:8080 -v <config.path>:/opt/cs-kv-storage/application.conf <tag>

where <tag>, <path to jar with dependencies>, <port> and <config.path> should be replaced with actual values.

<path to jar with dependencies> can be a path in a file system or URL, e.g.

https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=com.bwsw&a=cs-kv-storage_2.12&c=jar-with-dependencies&v=1.0.1-SNAPSHOT

target/scala-2.12/cs-kv-storage-1.0.1-SNAPSHOT-jar-with-dependencies.jar

⁠API

• Storage operations⁠
• Storage history⁠
• Storage management⁠
• Storage health⁠

⁠Storage operations

All operations require a valid Secret-Key header.

• Get the value by the key⁠
• Get values by keys⁠
• Set the value for the key⁠
• Set values for keys⁠
• Remove the mapping by the key⁠
• Remove mappings by keys⁠
• List keys⁠
• Clear⁠

⁠Get the value by the key

⁠Request

GET /get/<storage UUID>/<key>

⁠Response

⁠Get values by keys

⁠Request

POST /get/<storage UUID>
Content-Type: application/json

[
    "key1",
    "key2",
    "key3"
]

⁠Response

⁠Body example

In the following example the mapping for the second key does not exist.

{
    "key1": "value1",
    "key2": null,
    "key3": "value3"
}

⁠Set the value for the key

⁠Request

PUT /set/<storage UUID>/somekey
Content-Type: text/plain

somevalue

⁠Response

⁠Set values for keys

⁠Request

PUT /set/<storage UUID>
Content-Type: application/json

{
    "key1": "value1",
    "key2": "value2",
    "key3": "value3"
}

⁠Response

⁠Body example

In the following example values for the first and third keys are set successfully.

{
    "key1": true,
    "key2": false,
    "key3": true
}

⁠Remove the mapping by the key

⁠Request

DELETE /delete/<storage UUID>/somekey

⁠Response

⁠Remove mappings by keys

⁠Request

POST /delete/<storage UUID>
Content-Type: application/json

[
    "key1",
    "key2",
    "key3"
]

⁠Response

⁠Body example

In the following example mappings for the first and third keys are deleted successfully.

{
    "key1": true,
    "key2": false,
    "key3": true
}

⁠List keys

⁠Request

GET /list/<storage UUID>

⁠Response

⁠Body example

[
    "key1",
    "key2",
    "key3"
]

⁠Clear

⁠Request

POST /clear/<storage UUID>

⁠Response

⁠Storage history

• Search and list history records⁠
• List history records⁠

⁠Search and list history records

Secret-Key header is mandatory.

⁠Request

GET /history/<storage UUID>

⁠Parameters

All parameters are optional.

* start and end parameters can be used separately. If both start and end parameters are specified history records with dates that are greater/equal to start and less/equal to end are returned.

** If both page and scroll parameters are specified scroll is used.

⁠Response

⁠Body example

For requests with page parameter

{
   "total":1000,
   "size":10,
   "page":1,
   "items":[
      {
         "key":"key",
         "value":"value",
         "operation":"set/delete/clear",
         "timestamp":1528442057000
      }
   ]
}

For requests with scroll parameter

{
   "total":1000,
   "size":10,
   "scrollId":"scroll id",
   "items":[
      {
         "key":"key",
         "value":"value",
         "operation":"set/delete/clear",
         "timestamp":1528442057000
      }
   ]
}

⁠List history records

⁠Request

POST /history
Content-Type: application/json

{
   "scrollId":"scroll id",
   "timeout": 60000
}

⁠Response

⁠Storage management

All operations require a valid Secret-Key header.

• Update storage TTL⁠
• Delete the storage⁠

⁠Update storage TTL

⁠Request

PUT /storage/<storage UUID>?ttl=<ttl>

⁠Response

⁠Delete the storage

⁠Request

DELETE /storage/<storage UUID>

⁠Response

⁠Storage health

⁠Check health

⁠Request

GET /health

⁠Parameters

⁠Response

⁠Status code

⁠Body

If detailed = false then response body is empty, otherwise results are in the response body in the format specified below and the content type is application/json:

{
   "status":"HEALTHY/UNHEALTHY",
   "checks":[
      {
         "name":"<name>",
         "status":"HEALTHY/UNHEALTHY",
         "message":"<message>"
      }
   ]
}