jez500/seleniumbase-scrapper repository overview

⁠SeleniumBase API

This project exposes an HTTP API for SeleniumBase, allowing you to fetch web page content and metadata via HTTP requests.

Its API is based on scrapper⁠ (which uses playwrite) and uses the same parameters and response format so it should be interchangeable with it.

⁠Running the Container

Start the container with the API server:

docker run -d -p 8000:8000 --name seleniumbase_scrapper jez500/seleniumbase-scrapper

The API will be automatically started and available on port 8000.

⁠Configuration

⁠Environment Variables

You can set default values for all API parameters using environment variables. This is useful for configuring the behavior of the API without changing query parameters for each request.

Example with environment variables:

docker run -d -p 8000:8000 \
  -e DEFAULT_CACHE=true \
  -e DEFAULT_FULL_CONTENT=false \
  -e DEFAULT_SCREENSHOT=false \
  -e DEFAULT_INCOGNITO=true \
  -e DEFAULT_TIMEOUT=60000 \
  -e DEFAULT_SLEEP=1000 \
  --name seleniumbase_scrapper jez500/seleniumbase-scrapper

All available environment variables:

• DEFAULT_CACHE (default: false)
• DEFAULT_CACHE_TTL (default: 3600 - cache time-to-live in seconds, 60 minutes)
• DEFAULT_FULL_CONTENT (default: false)
• DEFAULT_SCREENSHOT (default: false)
• DEFAULT_USER_SCRIPTS (default: empty)
• DEFAULT_USER_SCRIPTS_TIMEOUT (default: 0)
• DEFAULT_INCOGNITO (default: true)
• DEFAULT_TIMEOUT (default: 60000)
• DEFAULT_WAIT_UNTIL (default: domcontentloaded)
• DEFAULT_SLEEP (default: 0)
• DEFAULT_RESOURCE (default: empty, all resources allowed)
• DEFAULT_VIEWPORT_WIDTH (default: empty)
• DEFAULT_VIEWPORT_HEIGHT (default: empty)
• DEFAULT_SCREEN_WIDTH (default: empty)
• DEFAULT_SCREEN_HEIGHT (default: empty)
• DEFAULT_DEVICE (default: Desktop Chrome)
• DEFAULT_SCROLL_DOWN (default: 0)
• DEFAULT_IGNORE_HTTPS_ERRORS (default: true)
• DEFAULT_USER_AGENT (default: empty)
• DEFAULT_LOCALE (default: empty)
• DEFAULT_TIMEZONE (default: empty)
• DEFAULT_HTTP_CREDENTIALS (default: empty)
• DEFAULT_EXTRA_HTTP_HEADERS (default: empty)

⁠User Scripts

User scripts allow you to execute custom JavaScript code on the page after it loads but before article extraction begins. This is useful for:

• Removing advertisements
• Clicking cookie consent buttons
• Handling modal popups
• Customizing page content

To use user scripts:

1. Create your JavaScript files and place them in the api/user_scripts directory
2. Reference them in the API call using the user-scripts parameter

Example user script (api/user_scripts/example-remove-ads.js):

// Remove common ad elements
(function() {
    const adSelectors = ['.advertisement', '.ad-container', '.ads', '#ad'];
    adSelectors.forEach(selector => {
        document.querySelectorAll(selector).forEach(el => el.remove());
    });
})();

Usage:

curl -X GET "http://localhost:8000/api/article?url=https://example.com&user-scripts=example-remove-ads.js"

Multiple scripts can be specified separated by commas:

curl -X GET "http://localhost:8000/api/article?url=https://example.com&user-scripts=remove-ads.js,accept-cookies.js"

⁠API Endpoints

⁠GET /api/article

Fetches article content and metadata from a specified URL using SeleniumBase.

Parameters:

All parameters except url have default values that can be set via environment variables (with DEFAULT_ prefix).

⁠Scraper Settings

⁠Browser Settings

Examples:

Basic usage:

curl -X GET "http://localhost:8000/api/article?url=https://en.wikipedia.org/wiki/web_scraping"

With caching enabled:

curl -X GET "http://localhost:8000/api/article?url=https://example.com&cache=true"

With full content and screenshot:

curl -X GET "http://localhost:8000/api/article?url=https://example.com&full-content=true&screenshot=true"

With custom viewport and sleep:

curl -X GET "http://localhost:8000/api/article?url=https://example.com&viewport-width=1024&viewport-height=768&sleep=2000"

With user scripts:

curl -X GET "http://localhost:8000/api/article?url=https://example.com&user-scripts=example-remove-ads.js&user-scripts-timeout=1000"

With scroll for lazy-loading content:

curl -X GET "http://localhost:8000/api/article?url=https://example.com&scroll-down=1000&sleep=2000"

Response Fields:

The response to the /api/article request returns a JSON object containing the following fields:

Example Response:

{
  "id": "13cfc98ddfe0fd340fbccd298ada8c17",
  "url": "https://en.wikipedia.org/wiki/Web_scraping",
  "domain": "en.wikipedia.org",
  "title": "Web scraping - Wikipedia",
  "byline": null,
  "excerpt": null,
  "siteName": null,
  "content": "<article>...</article>",
  "textContent": "Web scraping - Wikipedia\nJump to content...",
  "length": 27104,
  "lang": "en",
  "dir": "ltr",
  "publishedTime": null,
  "fullContent": "<html>...</html>",
  "date": "2025-11-11T22:37:42.235424Z",
  "query": {
    "url": "https://en.wikipedia.org/wiki/Web_scraping"
  },
  "meta": {
    "og_title": "Web scraping - Wikipedia",
    "og_type": "website"
  },
  "resultUri": "api://article/13cfc98ddfe0fd340fbccd298ada8c17",
  "screenshotUri": null
}

Error Handling:

Error responses follow this structure:

{
  "detail": [
    {
      "type": "error_type",
      "msg": "some message"
    }
  ]
}

For detailed error information, consult the Docker container logs.

Response Codes:

• Success (200): Returns JSON with article data and metadata
• Error (400): Missing URL parameter
• Error (500): Failed to fetch URL

⁠GET /health

Health check endpoint to verify the API is running.

Example:

curl -X GET "http://localhost:8000/health"

Response:

{
  "status": "healthy",
  "service": "jez500/seleniumbase-scrapper"
}

⁠GET /

Root endpoint that provides API documentation.

Example:

curl -X GET "http://localhost:8000/"

Response:

{
  "service": "SeleniumBase API",
  "version": "1.0.0",
  "endpoints": {
    "/api/article": {
      "method": "GET",
      "description": "Fetch HTML content from a URL",
      "parameters": {
        "url": "The URL to fetch (required)"
      },
      "example": "/api/article?url=https://en.wikipedia.org/wiki/web_scraping"
    },
    "/health": {
      "method": "GET",
      "description": "Health check endpoint"
    }
  }
}

⁠Usage Examples

⁠Fetch Wikipedia Page

curl -X GET "http://localhost:8000/api/article?url=https://en.wikipedia.org/wiki/Python_(programming_language)"

⁠Fetch Any Website

curl -X GET "http://localhost:8000/api/article?url=https://www.example.com"

⁠Save HTML to File

curl -X GET "http://localhost:8000/api/article?url=https://www.example.com" -o output.html

⁠Features

• Headless Browser: Uses Chrome in headless mode for efficient scraping
• JavaScript Rendering: Fully renders JavaScript-heavy pages
• Undetected Mode: Uses SeleniumBase's undetected mode to bypass bot detection
• Error Handling: Proper error responses with meaningful messages
• Auto Cleanup: Automatically closes browser drivers after each request

⁠Technical Details

• Framework: Flask
• Browser: Chrome (headless)
• Port: 8000
• SeleniumBase Driver: UC mode enabled for better compatibility

⁠Container Management

⁠View Logs

docker logs seleniumbase_scrapper

⁠Stop Container

docker stop seleniumbase_scrapper

⁠Start Existing Container

docker start seleniumbase_scrapper

⁠Remove Container

docker rm -f seleniumbase_scrapper

⁠Interactive Mode

You can also run the container interactively while still having the API available:

docker run -it -p 8000:8000 --name seleniumbase_scrapper jez500/seleniumbase-scrapper

The API server will start automatically in the background, and you'll have access to a bash shell.

⁠Troubleshooting

⁠Check if API is running

curl http://localhost:8000/health

⁠View API logs inside container

docker exec -it jez500/seleniumbase-scrapper bash
# Then check for the API process
ps aux | grep python

⁠Port already in use

If port 8000 is already in use, map to a different port:

docker run -d -p 9000:8000 --name seleniumbase_scrapper jez500/seleniumbase-scrapper
curl -X GET "http://localhost:9000/api/article?url=https://example.com"