A REST API for accessing NC Community Colleges WebAdvisor course information. This API provides programmatic access to course catalogs, subject categories, and detailed course information from the WebAdvisor system using Browserless for reliable web scraping.
GET / - API information and available endpointsGET /health - Health check endpointGET /docs - Interactive API documentation (Swagger UI)GET /redoc - Alternative API documentation (ReDoc)GET /api/subjects - Get all available subject categoriesGET /api/subjects/{subject_code}/courses - Get all courses for a specific subjectGET /api/courses/{subject_code}/{course_number} - Get detailed information for a specific courseGET /api/search?query={term}&limit={number} - Search courses across all subjectsGET /api/cache/info - Get cache statistics and informationDELETE /api/cache/clear - Clear all cached data (admin endpoint)Clone or download the project
cd webadvisor-api
Create a virtual environment
python -m venv venv
# On Windows
venv\Scripts\activate
# On macOS/Linux
source venv/bin/activate
Install dependencies
pip install -r requirements.txt
Configure environment variables
cp .env.example .env
# Edit .env file with your Browserless settings
Required Environment Variables:
BROWSERLESS_URL: Your Browserless service URL (e.g., https://chrome.browserless.io)BROWSERLESS_TOKEN: Your Browserless API tokenRun the API
# Development mode
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
# Production mode
uvicorn app.main:app --host 0.0.0.0 --port 8000
The API will be available at http://localhost:8000
BROWSERLESS_URL=https://chrome.browserless.io and BROWSERLESS_TOKEN=your_token in your .env filedocker run -p 3000:3000 browserless/chrome
BROWSERLESS_URL=http://localhost:3000 in your .env filecurl http://localhost:8000/api/subjects
Response:
{
"subjects": [
{
"code": "ACC",
"name": "Accounting"
},
{
"code": "AGR",
"name": "Agriculture"
}
],
"total_subjects": 2,
"last_updated": "2025-01-11T15:00:00Z"
}
curl http://localhost:8000/api/subjects/ACC/courses
Response:
{
"subject_code": "ACC",
"courses": [
{
"subject_code": "ACC",
"course_number": "120",
"course_title": "Principles of Financial Accounting",
"credits": 4.0,
"description": "This course introduces basic financial accounting concepts...",
"prerequisites": ["MAT 143 or equivalent"],
"corequisites": [],
"lecture_hours": 3.0,
"lab_hours": 2.0,
"work_experience_hours": 0.0,
"course_type": "Curriculum",
"transferability": "Transfers to UNC system",
"last_updated": "2025-01-11T15:00:00Z"
}
],
"total_courses": 1,
"last_updated": "2025-01-11T15:00:00Z"
}
curl "http://localhost:8000/api/search?query=accounting&limit=10"
curl http://localhost:8000/api/courses/ACC/120
The API can be configured using environment variables. Copy .env.example to .env and modify as needed:
BROWSERLESS_URL: Browserless service URLBROWSERLESS_TOKEN: Browserless API token (if using cloud service)API_HOST: Host to bind the API (default: 0.0.0.0)API_PORT: Port to run the API (default: 8000)CACHE_TTL_SUBJECTS: Cache TTL for subjects in seconds (default: 86400)CACHE_TTL_COURSES: Cache TTL for courses in seconds (default: 3600)RATE_LIMIT_MAX_REQUESTS: Max requests per window (default: 100)RATE_LIMIT_WINDOW_SECONDS: Rate limit window in seconds (default: 3600)The API includes comprehensive error handling:
webadvisor-api/
├── app/
│ ├── __init__.py
│ ├── main.py # FastAPI application and routes
│ ├── models.py # Pydantic data models
│ ├── scraper.py # WebAdvisor scraping logic with Browserless
│ ├── cache.py # Caching implementation
│ └── rate_limiter.py # Rate limiting implementation
├── requirements.txt # Python dependencies
├── .env.example # Environment configuration template
└── README.md # This file
# Install test dependencies
pip install pytest pytest-asyncio httpx
# Run tests (when implemented)
pytest
This repository is configured with GitHub Actions to automatically build and publish Docker images to Docker Hub as vfraziers/ncccs-api.
The GitHub Actions workflow automatically:
To enable automatic Docker Hub publishing, configure these secrets in your GitHub repository settings:
DOCKERHUB_USERNAME: Your Docker Hub usernameDOCKERHUB_TOKEN: Your Docker Hub access token (not password)DOCKERHUB_TOKEN secretOnce published, you can use the Docker image:
# Pull and run the latest image
docker pull vfraziers/ncccs-api:latest
docker run -p 8000:8000 --env-file .env vfraziers/ncccs-api:latest
# Or use docker-compose with the published image
# Update docker-compose.yml to use: image: vfraziers/ncccs-api:latest
For local development or manual deployment:
# The existing Dockerfile builds the image
docker build -t webadvisor-api .
docker run -p 8000:8000 --env-file .env webadvisor-api
# Production settings
API_DEBUG=False
BROWSERLESS_URL=https://chrome.browserless.io
BROWSERLESS_TOKEN=your_production_token
RATE_LIMIT_MAX_REQUESTS=1000
RATE_LIMIT_WINDOW_SECONDS=3600
When using Browserless cloud service:
This project is provided as-is for educational and development purposes.
This API scrapes data from the NC Community Colleges WebAdvisor system. Please use responsibly and in accordance with the website's terms of service. The API includes rate limiting to prevent overwhelming the source system.
For issues, questions, or contributions, please refer to the project documentation or create an issue in the project repository.
Browserless Connection Errors
BROWSERLESS_URL and BROWSERLESS_TOKENScraping Failures
Performance Issues
Enable debug logging by setting LOG_LEVEL=DEBUG in your .env file for detailed request/response information.
Content type
Image
Digest
sha256:d260d6641…
Size
69.1 MB
Last updated
over 1 year ago
docker pull vfraziers/ncccs-api