terorie/od-database-crawler repository overview

⁠OD-Database Crawler 🕷

• Crawler for OD-Database⁠
• In production at https://od-db.the-eye.eu/⁠
• Over 880 TB actively crawled
• Crawls HTTP open directories (standard Web Server Listings)
• Gets name, path, size and modification time of all files
• Lightweight and fast

https://od-db.the-eye.eu/⁠

⁠Usage

⁠Deploys

1. With Config File (if config.yml found in working dir)

   • Download default config⁠
   • Set server.url and server.token
   • Start with ./od-database-crawler server --config <file>

2. With Flags or env

   • Override config file if it exists
   • --help for list of flags
   • Every flag is available as an environment variable: --server.crawl_stats ➡️ OD_SERVER_CRAWL_STATS
   • Start with ./od-database-crawler server <flags>

3. With Docker

   docker run \
       -e OD_SERVER_URL=xxx \
       -e OD_SERVER_TOKEN=xxx \
       terorie/od-database-crawler
   

   Copy

⁠Flag reference

Here are the most important config flags. For more fine control, take a look at /config.yml.