A harvester for flickr content as part of Social Feed Manager.
For information on development and running tests, see the development documentation.
When running tests, provide Flickr credentials either as a
test_config.py file or environment variables (
FLICKR_SECRET). An example
test_config.py looks like:
FLICKR_KEY = "acbdfdfe6b8bba356e8ef278ed65dbbc8" FLICKR_SECRET = "16264549fc54cc33eb"
Running harvester as a service
Flickr harvester will act on harvest start messages received from a queue. To run as a service:
python flickr_harvester.py service <mq host> <mq username> <mq password>
Process harvest start files
Flickr harvester can process harvest start files. The format of a harvest start file is the same as a harvest start message. To run:
python flickr_harvester.py seed <path to file>
Iterating over photo records in a WARC
python flickr_photo_warc_iter.py <path to WARC>
Running exporter as a service
Flickr exporter will act on export start messages received from a queue. To run as a service:
python flickr_exporter.py service <mq host> <mq username> <mq password> <SFM UI REST API url>
Process export start files
Flickr exporter can process export start files. The format of an export start file is the same as an export start message. To run:
python flickr_exporter.py file <path to file> <SFM UI REST API url>
Harvest start messages
Following is information necessary to construct a harvest start message for the flickr harvester.
User harvest type
Api methods called:
- people.findByUsername to get nsid if username provided
- photos.getInfo for each photo
- username or nsid
- incremental: True (default) or False
- sizes: List of photo size labels. Default is Thumbnail, Large, and Original.
Extracted urls: Urls are generated for each photo for each size.