serenitystar/scraper repository overview

The Serenity* Star Scraper is a specialized web scraping tool designed to extract and process a wide range of web content, including HTML pages and RSS feeds. As a central feature of the Serenity* AI Hub Platform, this Dockerized tool is made available to users, clients, and the broader community to enhance the training of AI agents by transforming web data into actionable insights.

This solution simplifies the execution of web scraping tasks, eliminating the need for complex setup and dependency management. While the underlying code remains proprietary, the Docker image enables you to seamlessly integrate this tool into your local environment and data pipelines.

Key Use Cases:

• AI Training Data: Collect and preprocess datasets from various web sources to enhance the training and performance of AI models within Serenity* AI Hub.
• Content Aggregation: Gather and consolidate content from diverse websites to create comprehensive data feeds or reports that fuel AI-driven insights.
• Content Monitoring: Automate the tracking and analysis of web content updates, ensuring continuous enhancement of knowledge for AI agents.

How to Use:

1. Pull the Docker image:

   To get started, pull the latest Docker image from the Serenity* Star repository:

   docker pull serenitystar/scraper:latest
   

   Copy

2. Run the container:

   Execute the Docker container with your specified parameters to start scraping:

   docker run --rm -it \
     -v /path/to/local/output:/app/output \
     serenitystar/scraper:latest \
     --url "https://example.com" \
     -n 10 \
     --outputPath /app/output
   

   Copy
   • --url <URL>: (Required) The URL to start scraping from.
   • -c <ContentFileName>: (Optional) The name of the file where content will be saved. Defaults to content.md.
   • -n <MaxPages>: (Optional) The maximum number of pages to scrape. Defaults to 10.
   • --outputPath <OutputPath>: (Optional) The directory where the output files will be saved. Defaults to results.
   • --minLengthForEquality <MinLength>: (Optional) Minimum length of content to consider for equality when processing pages. Defaults to 50.
   • --separator <Separator>: (Optional) Separator string used between different content sections in the output file. Defaults to "%%%%%%%----////////\\\\\\\\\\\\\\\\----%%%%%%%.
   • --downloadableExtensions <Extensions>: (Optional) Comma-separated list of file extensions that should be downloaded during scraping. Defaults to .pdf,.doc,.docx,.txt,.csv,.xls,.xlsx,.ppt,.pptx.
   • --urlParameter <Parameter>: (Optional) URL parameter to add to every URL that is scraped.
   • --urlParameterValue <Value>: (Optional) Value for the URL parameter specified by --urlParameter.
   • --treatParametersAsDifferent <true|false>: (Optional) Treat URLs with different parameters as different pages. Defaults to true.

3. Check the output:

   After the scraping process is complete, check the output in your specified directory to review the extracted data.

Supported Platforms:

• Linux
• Windows
• macOS

License: MIT License

Documentation: Serenity* Star Docs (https://docs.serenitystar.ai⁠)

Support Channels: Join Serenity* Star Discord Channel! https://discord.com/invite/SrT3xP7tS8⁠

More info: Serenity* Star Ecosystem (https://serenitystar.ai⁠)

Copyright: 2024 - Subgen AI (https://subgen.ai⁠)