offs3c/gospiderwebui repository overview

⁠GoSpiderWebUI

GoSpiderWebUI is a web application for managing and viewing web crawling tasks.

⁠Prerequisites

• Node.js (v22 or later)
• npm
• Docker and Docker Compose (for containerized deployment)

⁠Setup

• Clone the repository:

  git clone https://github.com/OffS3c/gospiderwebui.git
  cd gospiderwebui
  

  Copy

• Install dependencies:

  npm install
  

  Copy

• Create a .env file in the root directory and add the following:

  REACT_APP_API_BASE_URL=http://localhost:3030/api/v1
  

  Copy

⁠Development

To run the application in development mode:

npm start

The application will be available at http://localhost:3000⁠.

⁠Production

To build the application for production:

npm run build

To run the production build:

npm run serve

⁠Docker Deployment

To build and run the application using Docker:

docker-compose up --build

The application will be available at http://localhost:3000⁠.

⁠API Integration

This application integrates with the following API endpoints:

• Add a Scan
• Cancel a Scan
• Delete a Scan
• List Scan Counts
• List Pending Scans
• List Running Scans
• List Completed Scans
• List Completed Scans with Limits
• List Cancelled Scans
• List Failed Scans
• Get Scan Info and Progress
• Retrieve Paginated Results

For more details on the API, please refer to the API documentation.

⁠Authentication

The application uses basic authentication. The default credentials are:

• Username: [email protected]⁠
• Password: admin

Make sure to change these credentials in a production environment.

⁠License

This project is licensed under the MIT License - see the LICENSE.md⁠ file for details.

⁠Next.js Setup

This project is also a Next.js⁠ application bootstrapped with create-next-app⁠.

⁠Getting Started

First, run the development server:

npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev

Open http://localhost:3000⁠ with your browser to see the result. You can start editing the page by modifying app/page.js. The page auto-updates as you edit the file.

⁠Learn More

To learn more about Next.js, take a look at the following resources:

• Next.js Documentation⁠ - Learn about Next.js features and API.
• Learn Next.js⁠ - An interactive Next.js tutorial.

You can check out the Next.js GitHub repository⁠ - your feedback and contributions are welcome!

⁠Deploy on Vercel

The easiest way to deploy your Next.js app is to use the Vercel Platform⁠ from the creators of Next.js. Check out our Next.js deployment documentation⁠ for more details.

⁠Environment Variables

⁠.env Configuration

# Application Port
UI_PORT=3000
APP_NAME=GO 🕷️ WEB UI
PORT=3030
WSS_PORT=3033

# Database Configuration
DB_USER=db_user
DB_PASSWORD=db_password
DB_HOST=db
DB_PORT=5432
DB_NAME=db_name

# Gospider Configuration
GOSPIDER_THREADS=2
GOSPIDER_DEPTH=3
GOSPIDER_CONCURRENCY=6
GOSPIDER_USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36"
GOSPIDER_TIMEOUT=10
GOSPIDER_BLACKLIST=''
GOSPIDER_JS_ENABLED='false'
GOSPIDER_SITEMAP_ENABLED='true'
GOSPIDER_ROBOTS_ENABLED='true'
GOSPIDER_OTHER_SOURCE_ENABLED='false'

# Other Configurations
EXTERNAL_ENDPOINT_FOR_SENDING_SCAN_STATUS_UPDATE=
MAX_SLOTS=2
MAX_LINKS=250000
MAX_TIME_IN_HRS=2
MIN_AVAILABLE_MEMORY=400

⁠.env.local Configuration

NEXTAUTH_SECRET=Ml6CbmuDU+YOUR+SECRECT+HERE+vVcyrVBtjHVCdZ+xV=
NEXTAUTH_URL=http://127.0.0.1:3000
UI_PORT=3000
API_BASE_URL=http://gospiderweb:3030
APP_NAME=GO 🕷️ WEB UI