vmandic/tris repository overview

⁠Tris - Simple Web Crawler

Github: https://github.com/vmandic/tris-simple-spider-scraper⁠

Simple Spider Scraper or Tris is a Node JS tool that is by its core feature a web crawler that allows you to recursively crawl a website and collect links within the specified depth. It provides various customization options to tailor the scraping process according to your needs.

Whether you're a developer, SEO professional, or data enthusiast, Tris provides a simple yet powerful tool to gather valuable insights from websites.

⁠Who Can Benefit?

⁠Developers

Tris is ideal for developers who need a quick and reliable way to extract links from a website, whether for indexing purposes, link analysis, or content mapping.

⁠SEO Professionals

SEO professionals can leverage Tris to gather valuable data about a website's structure, internal linking, and potential SEO opportunities.

⁠Data Enthusiasts

Data enthusiasts seeking to explore and analyze the structure of websites can use Tris to collect link data and gain insights into a website's content hierarchy.

⁠Features

• Customizable Settings: Configure the scraper with various settings using environment variables.
• Timeout Handling: Specify the timeout in milliseconds for each request.
• Path Depth Limitation: Set the maximum depth of paths to be scraped.
• Randomized User Agents: Provide a list of custom user-agent headers that are randomized between requests.
• Skip Words: Skip links that contain specified skip words.
• Sorting Output: Optionally sort the output file lines in ascending order.
• Delay Between Requests: Introduce a delay between requests to avoid overloading the server.
• HTTP Status Codes: Optionally include HTTP status codes in the output file.
• Include/Exclude Paths: Filter links based on specified path patterns.
• Trim Ending Slash: Control whether trailing slashes are removed from URLs.
• Exclude Query String and Fragment: Optionally exclude query strings and fragments from URLs.
• Limit amount of requests: Optionally limit the total amount of web requests to be sent.

⁠Why Tris?

Tris stands out as a simple yet effective solution for web scraping, providing a balance between customization and ease of use. As said in the beginning of this document, whether you're a developer, SEO professional, or data enthusiast, Tris empowers you to gather valuable insights from websites with minimal setup and maximum flexibility.

Start exploring the web with Tris today!