Public | Automated Build

Last pushed: 22 days ago
Short Description
App which ingests concepts from the kafka queue
Full Description

Concept Ingester

An API for reading concepts off of the kafka queue and sending them to the appropriate neo4j writer

An elasticsearch writer can also be configured. If so, after a successful ingestion into neo4j, the ingester will ingest the concept into elasticsearch too.
Incremental counters both for neo and elasticsearch are configured.


  • go get
  • cd $GOPATH/src/
  • go install

Running in a cluster

  • $GOPATH/bin/concept-ingester --services-list="people-rw-neo4j-blue,organisations-rw-neo4j-blue" --port="8081" --vulcan_addr="http://localhost:8080" --consumer_group_id="TestConcepts" --consumer_autocommit_enable=true --topic="Concept" --consumer_offset="smallest" --consumer_queue_id="kafka" --throttle=10

Some comments about configuration parameters:

  • --vulcan_addr the vulcan address, host and port
  • --services-list comma separated list of neo4j writers - do not append a port for running in the cluster
  • --elastic-service elasticsearch writer name
  • --topic, --consumer_group_id, --consumer_autocommit_enable, --consumer_offset, --consumer_queue_id see the message-queue-gonsumer library
  • the --consumer_queue_id/QUEUE_ID is used as a switch between clusters with vulcan-routing and those without - if this param is set, we assume vulcan-based routing.


  • Check connectivity http://localhost:8080/__health
  • Good to Go: http://localhost:8080/__gtg


How to run locally not in the cluster

concept-ingester --services-list="alphaville-series-rw-neo4j:8092" --port="8089" --vulcan_addr="http://localhost:8082" --consumer_group_id="alphaville-series" --topic="Concept" --consumer_of fset="smallest" --consumer_queue_id="kafka"
To run locally against local writers, specify the port each of these writers are running on (the writer address will be resolved to localhost:[specified-port]). Override the vulcan-addr parameter to point to the host:port of a kafka-rest-proxy.

Docker Pull Command

Comments (0)