Sign inSign up

coco/content-rw-elasticsearch

By coco

•Updated 11 days ago

Indexes V2 content in Elasticsearch for use by SAPI v1

Image
1

10K+

coco/content-rw-elasticsearch repository overview

⁠content-rw-elasticsearch

Circle CIGo Report Card Coverage Status

⁠Introduction

Indexes V2 content in Elasticsearch for use by SAPI V1

⁠Installation

Download the source code, dependencies and test dependencies:

    go get -u github.com/Financial-Times/content-rw-elasticsearch
    cd $GOPATH/src/github.com/Financial-Times/content-rw-elasticsearch
    dep ensure -vendor-only
    go test ./...

⁠Running locally

  1. Run the tests and install the binary:

     go test -race ./...
     go install
    
  2. Run the binary (using the help flag to see the available optional arguments):

     $GOPATH/bin/content-rw-elasticsearch [--help]
    

Options:

    --app-system-code="content-rw-elasticsearch"            System Code of the application ($APP_SYSTEM_CODE)
    --app-name="Content RW Elasticsearch"                   Application name ($APP_NAME)
    --port="8080"                                           Port to listen on ($APP_PORT)
    --aws-access-key=""                                     AWS ACCES KEY ($AWS_ACCESS_KEY_ID)
    --aws-secret-access-key=""                              AWS SECRET ACCES KEY ($AWS_SECRET_ACCESS_KEY)
    --elasticsearch-sapi-endpoint="http://localhost:9200"   AES endpoint ($ELASTICSEARCH_SAPI_ENDPOINT)
    --index-name="ft"                                       The name of the elaticsearch index ($ELASTICSEARCH_SAPI_INDEX)
    --kafka-proxy-address="http://localhost:8080"           Addresses used by the queue consumer to connect to the queue ($KAFKA_PROXY_ADDR)
    --kafka-consumer-group="default-consumer-group"         Group used to read the messages from the queue ($KAFKA_CONSUMER_GROUP)
    --kafka-topic="CombinedPostPublicationEvents"           The topic to read the meassages from ($KAFKA_TOPIC)
    --kafka-header="kafka"                                  The header identifying the queue to read the messages from ($KAFKA_HEADER)
    --kafka-concurrent-processing=false                     Whether the consumer uses concurrent processing for the messages ($KAFKA_CONCURRENT_PROCESSING)

3. Test:

There are no service endpoints to test.

⁠Build and deployment

⁠Healthchecks

Admin endpoints are:

`/__gtg`

Returns 503 if any if the checks executed at the /__health endpoint returns false

`/__health`

There are several checks performed:

  • Elasticsearch cluster connectivity

  • Elasticsearch cluster health

  • Elastic schema validation

  • Kafka queue topic check

    /__health-details

Shows ES cluster health details

`/__build-info` 

⁠Other information

An example of event structure is here testdata/exampleEnrichedContentModel.json⁠

The reference mappings for Elasticsearch are found here runtime/referenceSchema.json⁠

Tag summary

Content type

Image

Digest

sha256:fe92ec0fd…

Size

15.2 MB

Last updated

11 days ago

docker pull coco/content-rw-elasticsearch