exlskills/eocsutil repository overview

⁠EOCS Format Utilities

Command-line tool and GoLang API for working with EOCS and OLX courseware. This project also supports importing/updating course to EXLskills.com⁠ and e-books in PDF/MD formats!

⁠Prerequisites

• Linux or OS X recommended (PRs for Windows support are welcome)
• GoLang 1.10.^ (GOPATH must be set)
• GoLang dep tool Install Guide⁠
• NodeJS v8.10+
• Yarn 1.7+

⁠Installation

# Prior to running `go get`, make sure that you have setup Go with the $GOPATH environment variable, otherwise this will not work
go get -u github.com/exlskills/eocsutil
cd $GOPATH/src/github.com/exlskills/eocsutil
dep ensure -v
yarn install
go build # Optional, but this will validate that you have the correct golang deps

⁠Use of Git Commits for Last Updated Timestamps

The conversion process uses git commits of the source file structure to determine the last modification date for each course object. In the Server mode, the content of the git repository is cloned as part of the process. When loading a course from a local directory, if the directory is not a git clone, the assignment of timestamps will be bypassed

⁠Load EOCS course into EXLskills MongoDB and Elasticsearch (version 6.x)

⁠Assumptions

• EOCS-formatted course files are placed on the file system, preferably, in a git cloned local repository
• Target MongoDB is running locally (for a production/remote configuration, refer to your sysadmin/internal guides to get the mongodb connection URI; for MongoDB Atlas, use the 3.4 connection URI)

export MGO_DB_NAME="<name of the MongoDB target database>"

# Note: if Elasticsearch URI is not provided - the indexing will be bypassed
export ELASTICSEARCH_URI="http://localhost:9200"

# The default value for Elasticsearch "base" index is "learn". The actual index name will be set to the base plus "_<course launguage>", e.g., "base_en"
# To override the base name, set the Environment variable as below 
export ELASTICSEARCH_BASE_INDEX="learn"
 
# Note: `go run` will compile eocsutil on the fly with any code changes, to compile ahead of time, use `go build` and then execute the binary
# MongoDB URI *must* start with `mongodb:` - version 3.4 style
go run main.go convert --from-format eocs --from-uri <path to the course files folder> --to-format eocs --to-uri mongodb://localhost:27017

⁠Connecting to Elasticsearch HTTPS Backend in a non-production mode

Some Elasticsearch security models, e.g., AWS VPC Elasticsearch Service, require HTTPS connectivity in either Production or Test modes. To bypass Certificate validation in testing, ensure to set

export MODE=debug

A full example of connecting to AWS VPC Elasticsearch Service for testing:

// This is run in a separate terminal session that creates a tunnel to the AWS VPC via any instance running in the VPC:
ssh -i ~/.ssh/myAwsInstanceKey [email protected] -N -L 19200:vpc-my-es-service.us-west-2.es.amazonaws.com:443

// This is run in the eocsutil testing terminal session: 
export MODE=debug
export ELASTICSEARCH_URI="https://localhost:19200"

⁠Running in Server Mode

The server mode is design to automatically process course load from GitHub repositories into MongoDB upon receiving push notifications via GitHub Webhooks with application/json content type.

In addition to the Environment variables described above, these values should be configured:

• GH_SERVER_ADDR and GH_SERVER_PORT, 0.0.0.0 and 3344 by default - used launching the server
• GH_SERVER_MONGO_URI - course load destination, see the note on MongoDB URI above
• GH_USER_TOKEN and GH_WEBHOOK_SECRET - see GitHub Webhook documentation on configuring Token on the account and Secret in the Webhook. The Token is used to push changes to the repository that can be made when the loader runs, as well as to pull from private repositories
• GH_WEBHOOK_BRANCH - one value or a comma-delimited list with no spaces defining the case-sensitive filter on the branch(es) of the GitHub repository in scope to trigger the processing. Default "master"
• GH_AUTOGEN_COMMIT_MSG - a string used when the supplementary files in the course content are updated by the loader with the assigned record key values and re-committed back to the GitHub repository. The value is matched to bypass processing of the Webhook generated by the auto-commit. Default "auto#gen"
• SMTP_FROM_NAME, SMTP_FROM_ADDRESS, SMTP_HOST, SMTP_CONNECTION_STRING, SMTP_USER_NAME, SMTP_PASSWORD - used to send a confirmation email on the load results to the email associated with the GitHub commit. Set SMTP_HOST to "" to bypass this functionality
• SERVER_NICKNAME - used in the emails to identify the Server. Default - EOCS_GH

To run the server:

go run main.go serve-gh-hook 

⁠Server Endpoint

/v1/github/repo-push-event

⁠Testing GitHub Webhooks Locally with ngrok

• Install ngrok⁠ on the local box and launch it with http 3344 options
• Put the ngrok-generated URL into the GitHub webhook (without the port), e.g., https://1234ab56.ngrok.io/v1/github/repo-push-event
• Start the server locally and submit/resubmit the webhook for testing

⁠FAQ

⁠I'm getting errors about showdownjs

Sometimes there's an issue with showdownjs nodejs service that eocsutil spawns for markdown<->html conversion not exiting after eocsutil exits. If this occurs, run ps -ax | grep "node --harmony showdownjs/server.js" and kill that process.

⁠Showdownjs port conflict

Internally, we use a REST API server on http://localhost:6222⁠ to communicate with showdownjs. We're working to select a port on the fly, but until that feature is complete, you will have to either (a) modify eocsutil to use a different port or (b) temporarily stop whatever process is using port 6222 on your machine.