Public | Automated Build

Last pushed: 4 months ago
Short Description
A kinda simple distributed python crawler.
Full Description

kscrawl

A kinda simple distributed python crawler.

##Environment Variables
Set REDIS_HOST to the location of your redis server.

##How to use

To print crawl state:
python -m crawl.default.state

To start crawler:
Run a docker image from here https://hub.docker.com/r/mmontagna/kscrawl/
or
python -m crawl.default

To start a crawl:
python -m crawl.default.start --urls http://example.com --depth 1

Docker Pull Command
Owner
mmontagna
Source Repository