Public | Automated Build

Last pushed: 6 days ago
Short Description
Flavor of RabbitMQ with management for Social Feed Manager. Data is written to /sfm-data/rabbitmq.
Full Description


Utilities for Social Feed Manager.

Most significantly:

  • BaseConsumer: Base class for consuming messages from Rabbit.
  • BaseHarvester: Base class for a harvester, allowing harvesting from a queue or from a file.
  • StreamConsumer: A consumer intended to control stream harvests using Supervisor.
  • warced: An entry/exit wrapper for warcprox.


git clone
cd sfm-utils
pip install -r requirements/requirements.txt

Note that requirements/requirements.txt references the latest releast of warcprox-gwu.
If you are doing development on the interaction between warcprox-gwu and
sfm-utils, using requirements/dev.txt. This uses a local copy (../warcprox)
of warcprox-gwu in editable mode.

Alternatively, it can be included as a dependency by adding the following to requirements.txt:

#Replace 0.1.0 with the version that you want.
#If using any of the consumers or harvesters
#If using BaseHarvester    

Unit tests

python -m unittest discover
Docker Pull Command
Source Repository