Sign inSign up

ziplineai/chronon-fetcher

By ziplineai

•Updated about 5 hours ago

Image wrapping up the fetcher service

Image
0

10K+

ziplineai/chronon-fetcher repository overview

⁠Chronon Feature Fetching Service

Docker image that allows you to run the Chronon Feature Service.

⁠Launching the Service

To fetch the latest image, run:

docker pull ziplineai/chronon-fetcher:latest

To run the service, you can use the following command:

docker run \
 -p 9000:9000 \
 ziplineai/chronon-fetcher:latest
⁠Additional Configuration & Environment Variables

There's some additional environment variables that the Docker container can be started up with (using the -e "VAR=VALUE" syntax)

Environment VariableDescriptionDefault Value
Google Cloud Configuration
GCP_PROJECT_IDGCloud Zipline BigTable project``
GOOGLE_CLOUD_PROJECTGCloud Zipline BigTable project``
GCP_BIGTABLE_INSTANCE_IDGCloud Zipline BigTable instance id``
GOOGLE_APPLICATION_CREDENTIALSPath to GCloud application credentials json file⁠``
GCP_BIGTABLE_INSTANCE_IDGCloud Zipline BigTable instance id``
Profiler Configuration
ENABLE_GCLOUD_PROFILERIf set to 'true', enables profiling service using cloud profiler⁠false
Service Metrics Configuration
CHRONON_METRICS_READEREither of 'http' or 'prometheus'.``
EXPORTER_OTLP_ENDPOINTIf using the 'http' metrics reader, exports metrics to the configured endpoint using OTLP over http⁠http://localhost:4318
CHRONON_PROMETHEUS_SERVER_PORTIf using the 'prometheus' metrics reader, exposes a Prometheus service endpoint⁠ on the configured port to export Chronon library metrics8905
VERTX_PROMETHEUS_SERVER_PORTIf using the 'prometheus' metrics reader, exposes a Prometheus service endpoint⁠ on the configured port to export Vert.x webservice metrics8906
Monitoring
FETCHER_OOC_TOPIC_INFOTopic string in the format of: kafka://my-topic-name/key1=value1/key2=value2. If configured, the service emits online offline consistency feature logging⁠ data to the given topic``

⁠Exporting Metrics

The Chronon library and Vert.x⁠ fetcher web service can be collect metrics using the OpenTelemetry⁠ standard. Currently, the service supports two ways of exporting metrics:

  1. OTLP over HTTP: Metrics can be exported (push) to an OTLP endpoint using the EXPORTER_OTLP_ENDPOINT environment variable.
  2. Prometheus: Metrics can be exposed on a Prometheus endpoint (pull) using the CHRONON_PROMETHEUS_SERVER_PORT and VERTX_PROMETHEUS_SERVER_PORT environment variables.

⁠Service profiling

The service can be profiled using the Google Cloud Profiler⁠ by setting the ENABLE_GCLOUD_PROFILER environment variable to true. The profiles are exported to the cloud profiler under the service name 'chronon-fetcher'.

⁠Service Endpoints

The service exposes the following HTTP endpoints:

VerbEndpointDescription
GET/pingHealth check endpoint
GET/configFetcher service configuration
GET/v1/joinsList of Chronon joins marked online
GET/v1/join/:name/schemaReturns a join schema payload (consists of join name, entity key schema as an Avro string, value schema as an Avro string, schema hash
POST/v1/fetch/groupby/:nameRetrieve features for a specific GroupBy
POST/v1/fetch/join/:nameRetrieve features for a specific Join
⁠Example Usage

Some example curl calls to the service are shown below. The fetch join and GroupBy endpoints support bulk lookups, so you can pass a list of entity keys to fetch features for multiple entities in a single request.

$ curl http://localhost:9000/v1/joins                                                            
{"joinNames":["search.ranking_listing","search.recommendation_v1"]}

$ curl http://localhost:9000/v1/join/search.ranking_listing/schema | jq
{
  "joinName": "search.ranking_listing",
  "keySchema": "...",
  "valueSchema": "...",
  "schemaHash": "c72b6d",
  ...
}

$ curl -X POST http://localhost:9000/v1/fetch/join/ranking_listing -H 'Content-Type: application/json' -d '[{"listing_id":"12345"}]' | jq
{
  "results": [
    {
      "status": "Success",
      "entityKeys": {
        "listing_id": "12345"
      },
      "features": {
        "my_feature_1": 1,
        "my_feature_2": 0.5
        ...
      }
    }
  ]
}

Tag summary

Content type

Image

Digest

sha256:ae1e5c7fb…

Size

1.3 GB

Last updated

about 13 hours ago

docker pull ziplineai/chronon-fetcher