madtomy/jobscraper-api
FastAPI-based microservice for scraping and storing jobs
Image
Web servers
0
6.4K
madtomy/jobscraper-api repository overview
🧠 JobScraper API
A FastAPI-based microservice for scraping and storing job listings from major job platforms like Indeed, LinkedIn, ZipRecruiter, and Google Jobs.
🚀 Quick Start
Run directly from Docker Hub:
docker run -d \
--name jobscraper-api \
-p 8000:8000 \
-e DB_HOST=127.0.0.1 \
-e DB_PORT=3306 \
-e DB_USER=jobs \
-e DB_PASSWORD=jobs_pw \
-e DB_NAME=jobsdb \
-e LOG_LEVEL=info \
-e LOG_JSON=true \
madtomy/jobscraper-api:latest
Access Swagger UI at: 👉 http://localhost:8000/docs
⚙️ Environment Variables
| Variable | Description | Default |
|---|---|---|
APP_PORT | Application port | 8000 |
LOG_LEVEL | Logging level (debug, info, warning, error) | info |
LOG_JSON | Output structured JSON logs | true |
DB_HOST | MySQL database host | – |
DB_PORT | MySQL port | 3306 |
DB_USER | MySQL username | – |
DB_PASSWORD | MySQL password | – |
DB_NAME | Database name | jobsdb |
DB_POOL_SIZE | SQLAlchemy pool size | 10 |
DB_POOL_MAX_OVERFLOW | Connection overflow limit | 20 |
🧩 Features
- 🔍 Multi-source job scraping
- 🗄️ Persistent MySQL storage (SQLAlchemy ORM)
- 📘 Auto-generated Swagger UI (
/docs) - ⚙️ Database migrations (Alembic)
- 🧾 Structured JSON logging (Loguru)
- 🐳 Fully containerized (Docker & Compose support)
🧱 Example Docker Compose
version: "3.9"
services:
api:
image: madtomy/jobscraper-api:latest
container_name: jobscraper-api
ports:
- "8000:8000"
environment:
APP_ENV: production
DB_HOST: 127.0.0.1
DB_PORT: 3306
DB_USER: jobs
DB_PASSWORD: jobs_pw
DB_NAME: jobsdb
LOG_JSON: "true"
restart: unless-stopped
Start it:
docker compose up -d
🧠 Example API Usage
POST /scrape
Trigger a new scraping task:
{
"site_name": ["indeed", "linkedin", "google"],
"search_term": "software engineer",
"google_search_term": "software engineer jobs near Berlin Germany since yesterday",
"location": "Berlin",
"results_wanted": 20,
"hours_old": 72,
"country_indeed": "Germany",
"linkedin_fetch_description": true
}
🧾 License
Released under the MIT License © 2025 Madev
Tag summary
Content type
Image
Digest
sha256:7b575fb0b…
Size
440.8 MB
Last updated
about 1 month ago
docker pull madtomy/jobscraper-api