ioteam/scraper repository overview

⁠Scraper

This repository is a dockerised image of a Python web crawler app,The image is hosted on the Docker Hub and can be found here⁠.

⁠Technologies

Below are the main technologies used for this project. Take some time to familiarise yourself.

scrapy⁠: an open source and collaborative framework for scraping data from websites.

scrapyd⁠: a web service for your Scrapy spiders. Allows deployment and control of spiders via HTTP Json API.

scrapyd-client⁠: a client for scrapyd. Provides the scrapyd-deploy utility needed to deploy your spider projects to the Scrapyd server.

scrapy-splash⁠: Provides Scrapy + JavaScript integration using Splash.

scrapyrt⁠: Allows easy integration of HTTP API to your existing Scrapy project.

pillow⁠: A Python Imaging Library that supports the ImagesPipeline.

⁠How to run

First, you need to install and configure Docker on your system following this installation guide⁠.

Once Docker is successfully installed and configured on your system (you should be able to run $ docker run hello-world), you are ready to download a copy of this docker image.

⁠Downlaod local copy

Download the Docker image by using the following (replace :version with desired tag or empty for latest).

$ docker pull ioteam/scraper[:version]

According the docker-compose file, the image can be run as a background-daemon for scrapyd and as an interactive-shell for scrapy

⁠docker-compose.yml

scrapyd:
  image: ioteam/scraper:[version]
  ports:
    - "6800:6800"
  volumes:
    - ./data:/var/lib/scrapyd
    - /usr/local/lib/python2.7/dist-packages
  restart: always

scrapy:
  image: ioteam/scraper:[version]
  command: bash
  volumes:
    - ./data:/var/lib/scrapyd
    - .:/code
  working_dir: /code
  restart: always

⁠Run web server

Run the image as a background-daemon for scrapyd.

$ docker-compose up -d scrapyd

On your prefered browser, go to http://0.0.0.0:6800⁠ and make sure that your custom scraper appears in the list of projects (top of page: Available projects).

⁠Stop web server

To stop the web server (background-daemon) you need the id of the container (obtained by $ docker ps) and excecute the following: Remember to replace docker_container_id.

docker stop [docker_container_id]

⁠Clean-up

Current project will use Docker containers, images and volumes. Check out this⁠ handy cheat sheet on how to clean-up after you finished.