Public | Automated Build

Last pushed: 2 years ago
Short Description
Weibo harvester for Social Feed Manager.
Full Description

sfm-weibo-harvester

A basic harvester for Sina Weibo public post data as part of Social Feed Manager.

Provides harvesters for Sina Weibo API and Html Parser

Harvesting is performed by weibowarc.

The weibowarchtml is just a possible solution for any exist limitation problem. If possible, the weibo harvester will rely mostly on the API calling.

Install

git clone https://github.com/gwu-libraries/sfm-weibo-harvester
cd sfm-weibo-harvester
pip install -r requirements.txt

Testing

  • Install Docker and Docker-Compose
  • Geting the API_KEY, API_SECRET, REDIRECT_URI, ACCESS_TOKEN, refer to weibowarc
  • Start up the containers
    docker-compose -f docker/dev.docker-compose.yml up -d
    
  • Running the tests
    docker exec sfmdocker_sfmweiboharvester_1 python -m unittest discover
    
  • Check the logs
    docker-compose -f docker/dev.docker-compose.yml logs
    
  • Shutdown all the containers and clear what you have done
    docker-compose -f docker/dev.docker-compose.yml kill
    docker-compose -f docker/dev.docker-compose.yml rm -v --force
    docker rmi $(docker images -q)
    

Harvest start messages

The necessary information to construct a harvest start message for the weibo harvester.

Search friend timeline harvest type

Type

  • weibo_timeline

API called

  • statuses/friends_timeline

Optional parameters

  • incremental: True (default) or False

Summary

  • count for weibos
Docker Pull Command
Owner
victtan

Comments (0)