chiayinchen/medium-crawler repository overview

$ pipenv install

$ pipenv install --dev

Syntax: scrapy crawl medium <arguments>

Pass arguments to spider using the `-a` option.

Supported arguments:
    usernames: comma-separated writer's profile page names
    date: crawling date (YYYYMMDD)
    back: number of days to be crawled
    urls: comma-separated url list

* If `urls` is set, `usernames` will be ignored.
* If `date` is set, `back` will be ignored.

# Retrieve data from certain writer's profile pages and a specific date on
$ scrapy crawl medium -a usernames=chiayinchen -a date=20180801

# Retrieve data from certain urls
$ scrapy crawl medium -a urls=https://medium.com/8045c82962e2/be290cd1f9d8

$ python medium_crawler/run.py --spider medium

$ tox

$ docker pull chiayinchen/medium-crawler:<version>

$ docker run --rm -it -d --name <container_name> chiayinchen/medium-crawler:<version>

$ docker exec -it <container_name> sh