Public | Automated Build

Last pushed: 23 days ago
Short Description
App which ingests concepts from the kafka queue
Full Description

Concept Ingester

An API for reading concepts off of the kafka queue and sending them to the appropriate neo4j writer

An elasticsearch writer can also be configured. If so, after a successful ingestion into neo4j, the ingester will ingest the concept into elasticsearch too.
Incremental counters both for neo and elasticsearch are configured.


  • go get
  • cd $GOPATH/src/
  • go install

Running in a cluster

  • $GOPATH/bin/concept-ingester --services-list="people-rw-neo4j-blue,organisations-rw-neo4j-blue" --port="8081" --vulcan_addr="http://localhost:8080" --consumer_group_id="TestConcepts" --consumer_autocommit_enable=true --topic="Concept" --consumer_offset="smallest" --consumer_queue_id="kafka" --throttle=10

Some comments about configuration parameters:

  • --vulcan_addr the vulcan address, host and port
  • --services-list comma separated list of neo4j writers - do not append a port for running in the cluster
  • --elastic-service elasticsearch writer name
  • --topic, --consumer_group_id, --consumer_autocommit_enable, --consumer_offset, --consumer_queue_id see the message-queue-gonsumer library
  • the --consumer_queue_id/QUEUE_ID is used as a switch between clusters with vulcan-routing and those without - if this param is set, we assume vulcan-based routing.


This concept publishing pipeline is nearing end of life. It can currently only used to publish all organisations, factset people and authors. Most TME concepts have been switched over to use the new concept publishing pipeline described in detail here which are published via the basic-tme-transformer



How to run locally not in the cluster

concept-ingester --services-list="alphaville-series-rw-neo4j:8092" --port="8089" --vulcan_addr="http://localhost:8082" --consumer_group_id="alphaville-series" --topic="Concept" --consumer_of fset="smallest" --consumer_queue_id="kafka"
To run locally against local writers, specify the port each of these writers are running on (the writer address will be resolved to localhost:[specified-port]). Override the vulcan-addr parameter to point to the host:port of a kafka-rest-proxy.

Docker Pull Command