Exports concept from a data source (Neo4j) and sends it to S3
10K+
The service is used for automated concept exports. The concepts are taken from Neo4j, they are bundled into csv files and sent to S3 via UPP Export S3 Writer. There are 2 types of exports:
Download the source code, dependencies and test dependencies:
go get -u github.com/kardianos/govendor
go get -u github.com/Financial-Times/concept-exporter
cd $GOPATH/src/github.com/Financial-Times/concept-exporter
govendor sync
go build .
Run the tests and install the binary:
govendor sync
govendor test -v -race
go install
Run the binary (using the help flag to see the available optional arguments):
$GOPATH/bin/concept-exporter [--help]
Options:
Usage: concept-exporter [OPTIONS]
Exports concept from a data source (Neo4j) and sends it to S3
Options:
--app-system-code="concept-exporter" System Code of the application ($APP_SYSTEM_
CODE)
--app-name="concept-exporter" Application name ($APP_NAME)
--port="9090" Port to listen on ($APP_PORT)
--neo-url="http://localhost:7474/db/data" neo4j endpoint URL ($NEO_URL)
--s3WriterBaseURL="http://localhost:8080" Base URL to S3 writer endpoint ($S3_WRITER_B
ASE_URL)
--s3WriterHealthURL="http://localhost:8080/__gtg" Health URL to S3 writer endpoint ($S3_WRITER
_HEALTH_URL)
--conceptTypes=["Brand", "Topic", "Location", "Person", "Organisation"] Concept types to support ($CONCEPT_TYPES)
3. Test:
curl http://localhost:8080/__health
/export - Triggers an export. If conceptTypes is in the json body request, then a TARGETED export is triggered, otherwise a FULL exporte.g. A FULL export:
curl localhost:8080/__concept-exporter/export -XPOST
{"ID":"job_753c6005-dcf0-4381-96b9-aeac0d0c01c8","Concepts":["Brand","Topic","Location","Person","Organisation"],"Status":"Starting"}
A TARGETED export:
curl localhost:8080/__concept-exporter/export -XPOST -d '{"conceptTypes":"Brand Topic"}'
{"ID":"job_d6706835-5f72-4585-ba97-c454ea62dba6","Concepts":["Brand","Topic"],"Status":"Starting"}
/job - Returns the running job informatione.g.
curl http://localhost:8080/job | jq ''`
{
"ConceptWorkers": [
{
"ConceptType": "Brand",
"Count": 335,
"Progress": 335,
"Status": "Finished"
},
{
"ConceptType": "Topic",
"Count": 760,
"Progress": 760,
"Status": "Finished"
},
{
"ConceptType": "Location",
"Count": 13093,
"Progress": 13093,
"Status": "Finished"
},
{
"ConceptType": "Person",
"Count": 39668,
"Progress": 39668,
"Status": "Finished"
},
{
"ConceptType": "Organisation",
"Count": 80972,
"Progress": 80972,
"Status": "Finished"
}
],
"ID": "job_753c6005-dcf0-4381-96b9-aeac0d0c01c8",
"Concepts": [
"Brand",
"Topic",
"Location",
"Person",
"Organisation"
],
"Progress": [
"Brand",
"Topic",
"Location",
"Person",
"Organisation"
],
"Status": "Finished"
}
Admin endpoints are:
/__gtg
/__health
/__build-info
There are several checks performed:
Content type
Image
Digest
sha256:0ce35c3ae…
Size
11.7 MB
Last updated
2 months ago
docker pull coco/concept-exporter