schickling/crawl-notifier repository overview

⁠crawl-notifier

Crawl any website and get notified via mail when changes occur

⁠Features

• Deadsimple configuration & usage
• Runs inside a Docker⁠ container
• Uses jQuery for crawling
• Sends notifications via your Mailgun⁠ account
• Multiple receivers possible
• Runs until the check was successful

⁠Usage

$ docker run -d -e "MAILGUN_KEY=api-key" -v /my/local/crawlers:/data/crawlers schickling/crawl-notifier

For each page you want to crawl define a crawler file in a directory you mount with the -v argument above. A crawler file should be a UMD style module, defining an objects with receiver, sender, url and check properties like in the example below. The check method will be invoked with jQuery as parameter.

⁠Example Crawler

module.exports = {
	receiver: ['[email protected]', '[email protected]'],
	sender: '[email protected]',
	url: 'http://time.is',
	check: function($) {
		return $('#twd').text() === '23:00:00'; // bed time
	}
};

⁠Credits

• Node Mailgun API - https://github.com/shz/node-mailgun⁠
• Node Web Crawler - https://github.com/sylvinus/node-crawler⁠