offs3c/gospiderweb repository overview

⁠GOSPIDERWEB

⁠Overview

Gospiderweb is a web-based API for managing and interacting with Gospider scans. It allows you to initiate, monitor, and control web crawling scans through a RESTful API. This guide will help you set up and use the Gospiderweb API.

⁠Prerequisites

Before you begin, ensure you have the following installed on your system:

• docker
• curl
• jq
• websocat (for WebSocket interactions)

You can install these tools using the package manager for your distribution. For example, on Ubuntu, you can install them with:

sudo apt install docker.io curl jq websocat -y

⁠Installation

1. Clone the repository:

   git clone https://github.com/OffS3c/gospiderweb.git
   cd gospiderweb
   

   Copy

2. Copy the example environment configuration file:

   cp -f env.example .env
   

   Copy

3. Edit the .env file as needed:

   nano .env
   

   Copy

   You can optionally define an endpoint to which scan status updates will be sent by setting the EXTERNAL_ENDPOINT_FOR_SENDING_SCAN_STATUS_UPDATE variable in your .env file. For example:

   EXTERNAL_ENDPOINT_FOR_SENDING_SCAN_STATUS_UPDATE=http://your-endpoint-url.com/status-update
   

   Copy

4. Build and start the Docker containers:

   docker compose up --build -d
   

   Copy

5. View the logs to ensure everything is running correctly:

   docker compose logs -f
   

   Copy

⁠API Usage

Once the application is up and running, you can interact with the API using curl. Below are examples of various API requests:

⁠Add a Scan

curl -s -X POST http://127.0.0.1:3030/api/v1/scans/add \
-H "Content-Type: application/json" \
-d '{
  "url": "https://espn.com",
  "depth": 3,
  "external_scan_id": "123",
  "external_metadata": {
    "description": "espn.com crawl",
    "whatever_infos_you_want_to_reference_later": "bla bla bla"
  }
}' | jq -C

⁠Cancel a Scan

curl -s -X POST http://127.0.0.1:3030/api/v1/scans/cancel \
-H "Content-Type: application/json" \
-d '{
  "external_scan_id": "123"
}' | jq -C

⁠Delete a Completed OR Completed with Limits Scan

curl -s -X POST http://127.0.0.1:3030/api/v1/scans/delete \
-H "Content-Type: application/json" \
-d '{
  "external_scan_id": "123"
}' | jq -C

⁠List Scan Counts

curl -s -X GET http://127.0.0.1:3030/api/v1/scans/count | jq -C

⁠List Pending Scans

curl -s -X GET http://127.0.0.1:3030/api/v1/scans/list/pending | jq -C

⁠List Running Scans

curl -s -X GET http://127.0.0.1:3030/api/v1/scans/list/running | jq -C

⁠List Completed Scans

curl -s -X GET http://127.0.0.1:3030/api/v1/scans/list/completed | jq -C

⁠List Completed Scans with Limits

curl -s -X GET http://127.0.0.1:3030/api/v1/scans/list/completed-with-limits | jq -C

⁠List Cancelled Scans

curl -s -X GET http://127.0.0.1:3030/api/v1/scans/list/cancelled | jq -C

⁠List Deleted Scans

curl -s -X GET http://127.0.0.1:3030/api/v1/scans/list/deleted | jq -C

⁠List Failed Scans

curl -s -X GET http://127.0.0.1:3030/api/v1/scans/list/failed | jq -C

⁠Get Scan Info and Progress

curl -s -X GET http://127.0.0.1:3030/api/v1/scans/scan/123 | jq -C

⁠Retrieve Paginated Results

curl -s -X GET "http://127.0.0.1:3030/api/v1/scan/results/123?page=1&limit=10" | jq -C

⁠WebSocket Streaming

To stream real-time results from the Gospiderweb API using WebSocket, you can use websocat. This allows you to receive updates as they happen.

⁠Install websocat

If you haven't already installed websocat, you can do so by following the instructions provided here⁠.

⁠Start Streaming Results

Use the following command to start streaming results for a specific scan:

echo '{"fn":"getScanResultsStream","external_scan_id":"123"}' | websocat ws://127.0.0.1:3033

Replace 123 with your actual external_scan_id. This command will connect to the WebSocket server and begin streaming results for the specified scan.

⁠Scan Status Updates

You can configure Gospiderweb to send scan status updates to an external endpoint. To enable this feature, set the EXTERNAL_ENDPOINT_FOR_SENDING_SCAN_STATUS_UPDATE variable in your .env file.

When this feature is enabled, Gospiderweb will send updates in the following format to the specified endpoint:

{
  "external_scan_id": "123",
  "scan": {} // an object containing scan information including the status.
}

These updates will include real-time information about the scan status.

⁠License

This project is licensed under the GNU General Public License v2. See the LICENSE⁠ file for details.

⁠Contact

For any questions or issues, please open an issue on the GitHub repository⁠ or contact the project maintainer.