Public | Automated Build

Last pushed: 2 years ago
Short Description
Scrapyd docker image : import spiders from git, schedule with cron, expose files through http
Full Description


Scrapyd docker image

  • loads spiders(http crawlers) from local/git scrapy projects
  • customize spiders scheduling with cron syntax
  • files produced by exporters are exposed through http


  • Install docker


Here's a short explanation how to use scrapyd-docker:

  • in your scrapy project,

    • define EXPORT_PATH (e.g. EXPORT_PATH = "/tmp/export")
    • configure your exporters to write files in EXPORT_PATH folder (settings.get('EXPORT_PATH'))
    • when deployed to scrapyd, the scheduler will override this setting to make sure all exported files are stored in the correct http folder
  • Print the help :

      docker run --rm <gp3t1/scrapyd[:tag]> help
  • Run or create the container with the following command:

      docker <create|run [--rm] [-d] ...>
          -v </host/path/to/scrapyd/config>:/etc/scrapyd/
          -v </host/path/to/scrapyd/data>:/var/lib/scrapyd/
          -v </host/path/to/logs>:var/log/supervisor/
          -e SCRAPYD_USER=<scrapyd_user>
          -p <http_port>:8000
         [-p <scrapyd_api_port>:6800]
          --name <my-scrapyd>
    • <create|run [--rm] [-d] ...>: see docker run reference
    • </host/path/to/scrapyd/config>: your host path mapped to scrapyd config volume
    • </host/path/to/scrapyd/data>: your host path mapped to scrapyd data volume(populated with scrapyd logs, scrapy projects, exported files)
    • </host/path/to/logs>: your host path mapped to supervisor logs volume
    • <scrapyd_user>: start scrapyd as scrapyd_user (default='scrapyd')
    • <http_port>: available TCP port on your host mapped to scrapyd exported files
    • <scrapyd_api_port>: available TCP port on your host mapped to scrapyd api (not recommended)
    • --name <my-scrapyd>: name of the container to build
    • <gp3t1/scrapyd[:tag]>: image name (and tag) to build from
    • <scrapyd|help>: scrapyd container command. scrapyd launch scrapyd, the scheduler and the http server. help print the help
  • Configure your spiders

    • Spiders configuration file is found in your scrapyd data volume/spiders/spiders.json (create it before running the container or modify it after)
      format of the file:

            "spiders": {
                    "cron":"0 * * * *",
                    "cron":"0 8 * * 0,6",
      • <your-spider-id/name>: must be a valid linux folder name. It will be used to store/read your spidere in scrapyd data volume/spiders/<your-spider-id/name>
      • giturl: don't use it if you want scrapyd to laod your spider from scrapyd data volume/spiders/<your-spider-id/name>
      • cron: standard cron syntax used for your spider scheduling see for help
      • custom_settings: will be passed to scrapyd to override project settings during scheduling
      • custom_args: will be passed to scrapyd as arguments during scheduling
  • Administration

    • use docker <start|stop|restart|kill|logs> <my-scrapyd>. see docker command line
    • scrapyd configuration can be found in scrapyd config volume (for stability, some settings, like urls and paths, are forced during startup)
    • logs can be found in scrapyd data volume/logs and supervisor logs volume
    • the http folder containing exported files can be found in scrapyd data volume/spiders
    • use docker exec -ti <my-scrapyd> /bin/sh to get a terminal from your running container
    • In order to reload spiders configuration, you can
      • restart the container
      • restart the applications of the container with docker kill -s SIGHUP <my-scrapyd>
      • inside the container, execute supervisorctl start initSpiders
    • if you need to schedule a job manually, run su - <scrapyd_user> -c "runp /usr/local/bin/ crawl:<spider-id/name>" from inside the container


  1. Fork it
  2. Create your feature branch: git checkout -b feature/my-new-feature
  3. Commit your changes: git commit -am 'Add some feature'
  4. Push to the branch: git push origin feature/my-new-feature
  5. Submit a pull request

Requirements / Dependencies




Copyright (C) <year> <name of author>

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see

Docker Pull Command
Source Repository