anidata/palantiri repository overview

⁠Web Crawler for the Human Trafficking Project

This is the core web crawler that will be used for the human trafficking project

⁠Building

⁠Get the code

⁠Clone or Fork⁠

  # clone
  git clone [email protected]:atl-ads/palantiri.git     # ssh
  # or
  git clone https://gitlab.com/atl-ads/palantiri.git # http
  # build
  cd palantiri
  
  # Make sure you are using python3, then use pip to install dependencies
  # The anaconda package and version manager is easiest way to do this https://www.continuum.io/downloads
  pip install -e .

  # test
  python setup.py test

⁠Running

⁠Start a MongoMD or PostgreSQL Server

Install MongoDB or PostgreSQL and use the PostgreSQLDump⁠ or MongoDBDump⁠ class to store the collected data in a database.

⁠Scrape

  python search.py -[cgb] <site> <optional arguments>"

• -[cgb] defines the domain name. E.g. -b for .backpage.com
• site takes a comma separated list which defines the subdirectories to search. E.g. BusinessServices,ComputerServices
• optional arguments are defined with --<argument> value

A more detailed list may be obtained by running python search.py --help. example.py⁠ is an example of what we currently run. The run time for the program is around 30 minutes.

⁠More Documentation

• Running with Tor and Privoxy⁠
• Handling Signals to Shutdown Process⁠

⁠Dependencies

• Python 3.4⁠
• Selenium 2.47.1⁠
• Tor 2.7.4⁠
• Stem 1.4.1⁠
• Privoxy 3.0.23⁠

⁠Contributing

Please see CONTRIBUTING.md⁠ for more information about contributing to this project

⁠Questions

Please checkout our slack⁠ if you are already a part of the project or contact @danlrobertson if you have any questions.