sfm-elk works by listening for warc_created messages produced by harvesters. Upon receiving a message it:
- Iterates over social media API calls records recorded in the WARC using a platform-specific WARC iterator
outputting line-oriented json records to STDOUT.
- Pipes the json records to jq to filter out unnecessary metadata.
- Pipes the filtered json records to Logstash.
- Logstash loads the json records to Elasticsearch, which makes them available from Kibana.
sfm-elk currently supports loading WARCs from the Twitter REST API, Twitter Stream API, and Weibo API. A Twitter dashboard
and Weibo dashboard are provided.
For more information on deploying and using SFM ELK, see the Exploring Social Media Data with ELK.