[{"_1":2,"_161":-5,"_162":-5},"loaderData",{"_3":4,"_17":18,"_25":26,"_140":141},"root",{"_5":-5,"_6":-7,"_7":8,"_9":-7,"_10":11,"_12":-7,"_13":14,"_15":16},"currentUser","primaryEmail","siteVerification","6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI","toast","enableDev",false,"incidentStatus","theme","system","contrast","normal","routes/_layout",{"_19":-5,"_20":21,"_22":21,"_23":11,"_24":21},"username","isAskGordonEnabled",true,"isUsingNewTemplate","showMinimalLanding","showHoneypotTrap","routes/_layout.r.$namespace.$reponame",{"_27":28,"_29":-5,"_30":11,"_31":11,"_32":33,"_34":35,"_36":-5,"_37":11,"_38":39,"_88":-5,"_89":90,"_129":-5,"_130":11,"_131":21,"_132":11,"_133":-5,"_134":11,"_135":-5,"_136":137,"_138":91,"_139":-7},"canonicalUrl","https://hub.docker.com/r/davd/watchthatpage/tags","mcpServer","canSeeDockerfile","canSeeAutobuilds","namespace","davd","reponame","watchthatpage","cloudBuildSource","isAutobuildsAvailable","repo",{"_40":41,"_42":43,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":55,"_57":35,"_32":33,"_58":59,"_60":44,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":74,"_79":21,"_80":81,"_85":33,"_86":87},"categories",[],"content_types",[44],"image","date_registered","2018-01-16T11:25:09.661124Z","description","watchthatpage forked from https://github.com/SR-G/watchthatpage","is_private","last_updated","2018-04-13T23:45:11.829121Z","last_modified","2024-10-16T13:48:34.145251Z","media_types",[56],"application/vnd.docker.container.image.v1+json","name","pull_count",23998,"repository_type","star_count",0,"status",1,"status_description","active","source","collaborator_count","full_description","# watchthatpage\n\n`watchthatpage` is a command line program that may trigger mail notifications when some changes are detected on remote web pages. Monitored URLs are customizable (as many as you want) - in addition, some selectors may be configured in order to extract, if needed, only some relevant part. Notification mail are templated and are also configurable. Optionaly, some screenshots of website may be taken (with an external dependency, see examples).\n\n## Examples \n\n### Usage\n\n```bash\nWatchThatPage is a command line program used to trigger notifications when some HTML page contents is modified\n\nUsage:\n watchthatpage [command]\n\nAvailable Commands:\n clean Clean cached content\n grab Grab pages\n help Help about any command\n version Print the version number of watchthatpage\n\nFlags:\n --configuration string Configuration file name. Default is binary name + .json (e.g. 'watchthatpage.json'), in the same folder than the binary itself (default \"watchthatpage.json\")\n -h, --help help for watchthatpage\n\nUse \"watchthatpage [command] --help\" for more information about a command.\n```\n\nExample of output :\n\n```bash\nConfiguration file found under [watchthatpage.json], now loading content\nConfiguration loaded with [3] urls, gzip [false], minify [true], auto backup [true], generate screenshots [true], sections to skip [script footer meta style map img nav select form noscript]\nNow parsing URL [https://www.bostonglobe.com/news/bigpicture]\nNow parsing URL [https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Accueil_principal]\nNow parsing URL [https://news.google.com/news/?ned=fr&gl=FR&hl=fr]\nResults : \n - [DIFF] URL [https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Accueil_principal], analysis took [11.884853454s], cached content [/mnt/internal/sata/downloads/downloaded/workspaces/go/watchthatpage/bin/linux/cache//3f7b7021cd0f50958448273113c2ea1e]\n - [DIFF] URL [https://news.google.com/news/?ned=fr&gl=FR&hl=fr], analysis took [18.044830746s], cached content [/mnt/internal/sata/downloads/downloaded/workspaces/go/watchthatpage/bin/linux/cache//f3f1bce5db55b5e02b1479de568b0128]\n - [DIFF] URL [https://www.bostonglobe.com/news/bigpicture], analysis took [22.10838874s], cached content [/mnt/internal/sata/downloads/downloaded/workspaces/go/watchthatpage/bin/linux/cache//ee545bc3b80dc0cd684453067b12527a]\nTotal execution time [22.10883423s], analyzed urls [3], errors [0], diffs [3]\nNotifications to [......@gmail.com], from [webmaster@domain.tld], server [smtp.gmail.com:587], template [.../templates/multi-columns.tmpl]\n(...)\n```\n\nExample of generated mail :\n\n![Generated mail](resources/site/images/screenshot.png)\n\n### Configuration\n\nExample configuration :\n\n```json\n{\n \"Urls\" : [ \n\t\"https://asuswrt.lostrealm.ca/download\",\n\t\"http://tim.blog/gear/\"\n ],\n \"Selectors\" : {\n\t\"http://www.fiio.me/forum.php?mod=viewthread&tid=39932\" : { \"Selector\" : \"td[id=postmessage_105396]\", \"SelectorsToSkip\" : [ \"ignore_js_op\" ] }\n },\n \"LogLevel\" : \"INFO\",\n \"Gzip\" : false,\n \"MinifyHTML\" : true,\n \"GenerateScreenshots\" : true,\n \"ScreenshotCommand\" : \"/usr/bin/docker run --rm -v ${cache}:/images kevinsimper/wkhtmltoimage --quality 75 --crop-h 720 --format jpg ${url} /images/${filename}.jpg\",\n \"NotificationMail\" : { \n\t\"template\" : \"templates/multi-columns.tmpl\",\n\t\"to\" : \"@\",\n\t\"from\" : \"@\",\n\t\"subject\" : \"WatchThatPage results : {{ .NbDiff }} page(s) changed\",\n\t\"smtp-hostname\" : \"smtp.gmail.com\",\n\t\"smtp-tls\" : true,\n\t\"smtp-port\" : 587,\n\t\"smtp-login\" : \"@\",\n\t\"smtp-password\" : \"\"\n }\n ,\n \"SectionsToSkip\" : [ \n\t\"script\", \n\t\"footer\", \n\t\"meta\", \n\t\"style\", \n\t\"map\",\n\t\"img\",\n\t\"form\",\n\t\"noscript\"\n ]\n}\n```\n\n- **CacheDirectory** (optional, default is `cache/`) : defines where previous pages content will be stored\n- **Gzip** (optional, false by default) : defines if cached content has to be gzipped in order to reduce space usage\n- **MinifyHTML** (optional, false by default) : defines if retrieved HTML has to be minified (removal of whitespaces, ...)\n- **SectionsToSkip** (optional, empty by default) : list of HTML tags that needs to be removed (should include tags that may be different at each page load) : `script`, `footer`, `meta`, `form`, `noscript`, `style` are good values to exclude\n- **LogLevel** : not implemented at this time\n- **NotificationMail** : which parameters to use for the mail notification\n- **AutoBackup** (optional, false by default) : automatically creates a backup of the previous cache content (allows to manually review differences)\n- **Urls** : URLs to grab (without additional customizations)\n- **Selectors** : URLs to grab, but with additional selectors (to include one HTML entry to grab and/or to add exclusions)\n\n\t- **Selector** (optional) : the HTML entry to grab (example : `div[id=article_show_head_right]`) \n\t- **SelectorsToSkip** (optional, list of string) : list of entries to remove\n\n- **GenerateScreenshots** : (optional, false by default) should screenshots be generated or not - next parameter must then be configured\n- **ScreenshotCommand** (optional, blank by default) : system command that will be executed in order to take screenshot of remote web pages. Variables like `${url}`, `${cache}`, `${filename}` will be injected at running time and needs thus to be entered exactly like this in the JSON configuration. Examples may be :\n\n\t- execution through docker : `/usr/bin/docker run --rm --name wkhtmltoimage -v ${cache}:/images kevinsimper/wkhtmltoimage --quality 75 --crop-h 720 --format jpg ${url} /images/${filename}.jpg` (may be used exactly like that)\n\t- execution without docker : `wkhtmltoimage --quality 75 --crop-h 720 --format jpg ${url} /${filename}.jpg` (cache path has to be customized)\n\t\n### Template\n\nAvailable items are defined in the `results.go` file\n\n- **NbUrls** (int) : number of configured / analyzed urls\n- **NbDiff** (int) : numer of links containing differences with previous execution\n- **NbErrors** (int) : number of errors found \n- **ExecutionTime** (string) : duration of the whole execution\n- **Date** (string) : execution date\n- **Version** (string) : program version\n- **Results** (array or Result) : each individual result, whatever its state is (error, difference found, ...)\n\nAnd for each result (in the `result.go` file) :\n\n- **Url** (string) : the analyzed URL\n- **Title** (string) : page title\n- **Differences** (string) : description of found differences (in progress)\n- **FirstExecution** (bool) : is it the first execution (in this case, nothing can be analyzed)\n- **HasDifferences** (bool) : has this url any differences with previous execution\n- **HasError** (bool) : is the analysis in error\n- **Error** (string) : description of the encountered error\n- **CacheFileName** (string) : name of the stored cache content\n- **AnalysisExecutionTime** (time.Duration) : duration of the analysis of this url\n\nExample of a basic template (has to be configured in the `json` configuration file) :\n\n```html\n\n\n

\nOn {{ .Date }}, {{ .NbUrls }} URLs have been analyzed - {{ .NbErrors }} error(s), {{ .NbDiff }} difference(s), execution time {{ .ExecutionTime }}.
\n

\n\n

\nList of found differences :\n

{{ .Url }}

\n\n\n\n```\n\n### Crontab\n\nIn order to have this process running each day, just put in the system crontab something like :\n\n```crontab\n30\t06\t*\t*\t*\t\t/home/applications/watchthatpage/watchthatpage grab > /var/log/cron-watchthatpage.log 2>&1\n```\n\nFolder structure is dependant of the configuration (default configuration file in same folder - otherwise to be specified through the --configuration flag), templates path defined in JSON configuration.\n\n```bash\ncache\ntemplates\nwatchthatpage\nwatchthatpage.json\n```\n\n## Links \n\n- cobra framework : https://github.com/spf13/cobra\n- mail template borrowed from : https://github.com/InterNations/antwort\n- documentation about parsing HTML : \n\t- http://stackoverflow.com/questions/30109061/golang-parse-html-extract-all-content-with-body-body-tags\n\t- https://schier.co/blog/2015/04/26/a-simple-web-scraper-in-go.html\n\t- https://mionskowski.pl/html-golang-stream-processing\n\n## Development\n\n### Build\n\nInit and build from host :\n\n```bash\ndocker pull golang:alpine\ndocker run --rm -it -v $(pwd):/go golang go get -d ./...\ndocker run --rm -it -v $(pwd):/go golang go install tensin.org/watchthatpage\n```\n\nWork from inside a container :\n\n```bash\ndocker run --rm -it -v $(pwd):/go golang /bin/bash\n```\n\nBuild from alpine docker image (parameters are used to generate static and reduced binaries) : \n\n```bash\ngo install -ldflags \"-d -s -w -X tensin.org/watchthatpage/core.Build=`git rev-parse HEAD`\" -a -tags netgo -installsuffix netgo tensin.org/watchthatpage \n```\n\nCross-compile :\n\n```bash\nGOARCH=amd64 GOOS=windows go install ...\n```\n\n### Dependencies\n\n```\ngithub.com/PuerkitoBio/goquery\ngithub.com/fatih/color\ngithub.com/spf13/cobra\ngithub.com/tdewolff/minify\ngithub.com/tdewolff/minify/css\ngithub.com/tdewolff/minify/html\ngithub.com/tdewolff/minify/js\ngolang.org/x/net/html\n```\n\n```bash\ngrep -R -h \"github\" *|sort -u \ngrep -R -h \"golang\" *|sort -u \n```\n\n## TODO\n\n### To add\n\n- [x] Notifications : by mail\n- [x] Comparing content : option to minify read content\n- [x] Comparing content : configurable global exclusions (footer, header, ...)\n- [x] Optimization : option to gzip saved content\n- [x] Notifications : better default mail template\n- [x] Optimization auto backup previous cache content (if configured)\n- [x] Comparing content : optional selector for each url (\"body.section\", ...)\n- [x] Notifications : try to screenshot webpage and include image in template (seems hard in pure go : done with external command)\n- [ ] Notification : through MQTT\n- [ ] Project : add a \"serve\" mode (running server), exposing REST services to add/update/delete/trigger configured URLs\n- [ ] Comparing content : try to find a workaround for instagram pages\n- [ ] Comparing content : section / article detection (to be used instead of body)\n- [ ] Comparing content : try to compute real differences \n- [ ] Project : build, package and cross compile tooling \n- [ ] Project : code refactoring\n\t- [ ] use loggers\n\t- [ ] better separation of concerns between objects\n\t- [ ] add more tests\n\n### Find a proper name\n\nKeywords : \n\n```\nwatch\nwatch that page\nexplore\nanalyze\nscrap\nparse\ndiff\ndifferences\ndelta\nextract\nread\n```\n\n","has_starred","hub_user","immutable_tags_settings",{"_75":11,"_76":77},"enabled","rules",[78],".*","is_automated","permissions",{"_82":21,"_83":11,"_84":11},"read","write","admin","user","storage_size",3585092116,"orgData","recentTags",[91],{"_92":44,"_93":94,"_95":96,"_97":98,"_99":100,"_50":118,"_119":94,"_120":33,"_121":56,"_57":122,"_123":124,"_125":109,"_126":118,"_127":66,"_128":21},"content_type","creator",2053229,"full_size",198085543,"id",20691993,"images",[101],{"_102":103,"_104":105,"_106":107,"_108":109,"_110":111,"_112":113,"_114":107,"_115":-5,"_116":96,"_63":66,"_117":-5},"architecture","amd64","digest","sha256:dac5a5cdac2b2a2af2229498aa8f19adf7a1d954df72daa0f217815b3ef5fa69","features","","last_pulled","2026-10-01T07:26:49.195286325Z","last_pushed","2018-04-13T23:45:12Z","os","linux","os_features","os_version","size","variant","2018-04-13T23:45:11.434524Z","last_updater","last_updater_username","media_type","latest","repository",3525876,"tag_last_pulled","tag_last_pushed","tag_status","v2","sbxKitMetadata","isPotentialSpam","isHubSbxKitsEnabled","showSbxLogoOverride","sbxTrademarkText","displaySecurityMetadata","tagAttestations","tagsHaveAttestations",[],"selectedTag","attestationsUrl","routes/_layout.r.$namespace.$reponame.tags",{"_38":142,"_149":150,"_136":159,"_160":11},{"_40":143,"_42":144,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":145,"_57":35,"_32":33,"_58":59,"_60":44,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":146,"_79":21,"_80":148,"_85":33,"_86":87},[],[44],[56],{"_75":11,"_76":147},[78],{"_82":21,"_83":11,"_84":11},"tags",{"_151":64,"_152":-5,"_153":-5,"_154":155},"count","next","previous","results",[156],{"_92":44,"_93":94,"_95":96,"_97":98,"_99":157,"_50":118,"_119":94,"_120":33,"_121":56,"_57":122,"_123":124,"_125":109,"_126":118,"_127":66,"_128":21},[158],{"_102":103,"_104":105,"_106":107,"_108":109,"_110":111,"_112":113,"_114":107,"_115":-5,"_116":96,"_63":66,"_117":-5},[],"canDeleteTags","actionData","errors"]

davd/watchthatpage

Tags for davd/watchthatpage