Public | Automated Build

Last pushed: 10 months ago
Short Description
Proof-of-concept analytics dashboard for Social Feed Manager using ELK stack.
Full Description


A proof-of-concept analytics dashboard for social media content harvested by Social Feed Manager using the ELK
(Elasticsearch, Logstash,
Kibana) stack.

sfm-elk works by listening for warc_created messages produced by harvesters. Upon receiving a message it:

  1. Iterates over social media API calls records recorded in the WARC using a platform-specific WARC iterator
    (e.g., twitter_rest_warc_iter),
    outputting line-oriented json records to STDOUT.
  2. Pipes the json records to jq to filter out unnecessary metadata.
  3. Pipes the filtered json records to Logstash.
  4. Logstash loads the json records to Elasticsearch, which makes them available from Kibana.

sfm-elk currently supports loading WARCs from the Twitter REST API, Twitter Stream API, and Weibo API. A Twitter dashboard
and Weibo dashboard are provided.

For more information on deploying and using SFM ELK, see the Exploring Social Media Data with ELK.

Docker Pull Command
Source Repository