=======
Alpine based scrapyd image.
Scrapy is an open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.
Scrapyd is a service for running Scrapy spiders. It allows you to deploy your Scrapy projects and control their spiders using a HTTP JSON API.
pillow is the Python Imaging Library to support the ImagesPipeline.
Deploy and Control your spiders from host.Image does not include scrapyd-client nor scrapyd_api.
scrapyd:
image: raflman/scrapyd
ports:
- "6800:6800"
volumes:
- ./data:/scrapyd
- scrapy-packages-vol:/usr/lib/python3.6/site-packages
restart: always
scrapy:
image: raflman/scrapyd
command: sh
volumes:
- .:/code
- scrapy-packages-vol:/usr/lib/python3.6/site-packages
working_dir: /code
restart: always
$ docker-compose up -d scrapyd
$ pip install scrapyd-client
$ scrapy startproject myproject
$ cd myproject
$ scrapyd-deploy
File: scrapy.cfg
[settings]
default = myproject.settings
[deploy]
url = http://localhost:6800/
project = myproject
$ cat > quotes_spider.py << _EOF_
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'http://quotes.toscrape.com/tag/humor/',
]
def parse(self, response):
for quote in response.css('div.quote'):
yield {
'text': quote.css('span.text::text').extract_first(),
'author': quote.xpath('span/small/text()').extract_first(),
}
next_page = response.css('li.next a::attr("href")').extract_first()
if next_page is not None:
yield response.follow(next_page, self.parse)
_EOF_
$ docker-compose run --rm scrapy
>>> scrapy runspider quotes_spider.py -o quotes.json
>>> exit
Content type
Image
Digest
Size
41.3 MB
Last updated
about 6 years ago
docker pull raflman/scrapyd