Public Repository

Last pushed: 2 years ago
Short Description
Crawl domains from http://www.webhosting.dk for language=DKK, days limit = 28
Full Description

Crawl domains and save to MySql db

[ Usage ]

docker run -e USER_AGENT="Opera1" -e MYSQL_DB_HOST="172.17.0.2" -e MYSQL_DB_USER="scrapy" -e MYSQL_DB_PASSWORD="scrapy" -e MYSQL_DB_DATABASE="scrapy" -e INSTALL_DB="true" vdrizheruk/domains-scrapy

[ Environment variables ]

HOSTNAME - hostname

BOT_NAME = "domain" - Crawler bot name

USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)"

START_URL = "http://www.webhosting.dk/cgi-bin/domainscannerview.pl?language=DKK&sortby=2&showdayslimit=28"

ALLOWED_DOMAIN "webhosting.dk"

MYSQL_DB_HOST - db host

MYSQL_DB_USER - db user

MYSQL_DB_PASSWORD - db password

MYSQL_DB_DATABASE - db database name

INSTALL_DB - install db structure or use existed (already installed)

Docker Pull Command
Owner
vdrizheruk

Comments (0)