pratham1uk/letsnscrape-backend repository overview

⁠LetsNScrape — Backend API | v1.0.0

Async web scraping API built with FastAPI. Part of the NScrape series.

Supports both static and JavaScript-heavy websites using dual scraping engines — Scrapy/HTTP for fast static pages and Playwright (headless Chromium) for React, Vue, Angular, and Next.js SPAs. Auto-detects which engine to use. Jobs run asynchronously, results are cached in Redis, and completed data can be exported as PDF or DOCX.

---

⁠What this image does

• REST API served by FastAPI + Uvicorn on port 3070
• Scrapes any URL with configurable filters: text, images, links, metadata, headings, tables, or everything at once
• Auto-detects JS-heavy sites and switches to Playwright automatically
• Supports multi-page crawling with depth and domain controls
• Async job queue — start a scrape, poll for results
• Caches all jobs and results in Redis
• Exports scraped data as PDF or DOCX
• Swagger UI at /docs

---

⁠Prerequisites

• Docker Engine 20.10+
• At least 2GB RAM (Playwright/Chromium needs it)
• Redis container must be running before this starts

---

⁠⚠️ Mandatory companion containers

This image will NOT work standalone. You need all three:

1. redis:7.2-alpine — job store (required)
2. pratham1uk/letsnscape-backend:v1.0.0 — this image
3. pratham1uk/letsnscape-frontend:v1.0.0 — the UI

---

⁠Quick start

docker network create letsnscape-net

docker run -d
--name letsnscape-redis
--network letsnscape-net
redis:7.2-alpine
redis-server --port 6380

docker run -d
--name letsnscape-backend
--network letsnscape-net
-p 3070:3070
-e REDIS_HOST=letsnscape-redis
-e REDIS_PORT=6380
pratham1uk/letsnscape-backend:v1.0.0

API: http://localhost:3070⁠ Docs: http://localhost:3070/docs⁠

---

⁠Recommended: Docker Compose

Download the compose file and env template — no source code needed:

curl -O https://raw.githubusercontent.com/pratham1uk/letsnscape-public/main/docker-compose.prod.yml⁠ curl -O https://raw.githubusercontent.com/pratham1uk/letsnscape-public/main/.env.example⁠ cp .env.example .env docker-compose -f docker-compose.prod.yml up -d

---

⁠Environment variables

REDIS_HOST Redis container hostname (default: redis) REDIS_PORT Redis port (default: 6380) REDIS_DB Redis database index (default: 0) MAX_PAGES Max pages per job (default: 50) REQUEST_TIMEOUT HTTP timeout in seconds (default: 30) PLAYWRIGHT_TIMEOUT Playwright timeout in ms (default: 30000) DEBUG FastAPI debug mode (default: false)

---

⁠Port

3070 — FastAPI REST API (map to any host port: -p 8000:3070)

---

⁠API endpoints

POST /api/v1/scrape Start async scrape job GET /api/v1/scrape/{id} Poll status and fetch results POST /api/v1/export Download results as PDF or DOCX GET /api/v1/health Health check (includes Redis status) GET /docs Swagger UI

---

⁠Stack

FastAPI 0.115 | Uvicorn | Playwright 1.47 | Scrapy 2.11 BeautifulSoup4 | httpx | ReportLab 4.2 | python-docx 1.1 Redis asyncio | Pydantic 2 | Python 3.11

Part of the NScrape series by pratham1uk.