Public | Automated Build

Last pushed: 5 months ago
Short Description
Docker image for node script that scrapes the Oregon State course catalog
Full Description

scraper

Node.js script scrapes the entire Oregon State course catalog into a
MongoDB database.

Scraping into MongoDB

The scraper has two options for outputting class data. The first and default
option is to output data to a MongoDB database. This is accomplished by
running the following line:

npm start

The MongoDB instance this scrapes into can be modified with the MONGO_URL
environment variable (default is mongodb://localhost:27017/test). For
debugging, you can access the databse with the following command:

mongo localhost:27017/test

Scraping to command line or piping to other programs

Alternatively, the scraper can output course data straight to the command
line. This is accomplished by appending the --console flag when calling the
program like so:

npm start -- --console
Docker Pull Command
Owner
rutgerfarry
Source Repository