idrodriguez/scrapyd-auth repository overview

⁠Scrapyd with nginx authentication

Scrapy⁠ daemon is not developed as a daemon that allows authentication for access. Therefore, to protect your daemon, you need to serve scrapyd through a web server like nginx⁠. Bypassing the request to scrapyd internally and returning the output to nginx.

In this setup, nginx is working on port 80 and scrapyd in port 6800; externally, you will need to map only the port 80 as is mentioned in the docker-compose.yml⁠ file.

logparser⁠ is needed if you are using ScrapydWeb⁠ for managing your projects. It is included and running in the docker instance, communicating directly with scrapyd.

⁠.env file

You must setup your .env file accordingly to what your spider is requesting. For authentication purposes only, use:

USERNAME=USERNAME
PASSWORD=PASSWORD

After the setup you can reference in the docker-compose.yml⁠ file.

⁠DockerHub Images

You can find the image scrapyd-auth⁠ in dockerhub.

⁠Feedback

⁠Issues

If you have any problems with or questions about this image, please file an issue on the GitHub repository⁠.

⁠Contributing

Pull requests welcome :-)