Public | Automated Build

Last pushed: a month ago
Short Description
o2r muncher
Full Description

o2r muncher

Node.js implementation of the endpoints /api/v1/compendium (reading and metadata update) and /api/v1/jobs of the o2r-web-api.


  • Docker
  • MongoDB


This project includes a Dockerfile which can be built and run with the following commands.

docker build -t muncher .

docker run --name mongodb -d -p 27017:27017 mongo:3.4

docker run -it -p 8080:8080 --link mongodb:mongodb -v /var/run/docker.sock:/var/run/docker.sock -e MUNCHER_MONGODB=mongodb://mongodb:27017 -e DEBUG=muncher,muncher:* muncher


You can override these environment variables (configured in config/config.js) when starting the service to configure it.

    Define on which Port muncher should listen. Defaults to 8080.
    Location for the mongo db. Defaults to mongodb://localhost/. You will very likely need to change this.
    Which database inside the MongoDB should be used? Defaults to muncher.
    Base path for the compendia storage. Defaults to /tmp/o2r. If you want persistent compendia storage, you should point this to a separate volume.
    The name of the volume where compendia are stored, needed for mounting the correct path to 2nd level containers in compose configurations; overrides MUNCHER_BASEPATH for the metadata tools containers. Not set by default.
    User name or id for the user running the compendium containers, defaults to 1000. _Change this for usage with docker-compose!
    Email configuration settings for sending emails when critical events in the server happen, based on nodemailer. _TRANSPORT ist the mail transport string, see nodemailer documented, _RECEIVERS is a comma-separated list, and _SENDER is the mails sender. All three must be set. Mail notification can also be disabled completely via config.js.
    Docker image name and tag for metadata tools, defaults to running latest o2r-meta in a container, i.e. o2rproject/o2r-meta:latest.
    User name or id for the user running the meta container, defaults to o2r.
    Remove the metadata extraction and brokering containers after completion, defaults to true.
    Docker image name and tag for containerit tool, defaults to running Rocker's geospatial image with containerit pre-installed, i.e. o2rproject/containerit:geospatial.
    The user within the container, which must match the used image (see previous setting), defaults to rstudio, which is suitable for images in the rocker/verse stack of images. _Change this for usage with docker-compose!
    Should an error be thrown when files for a compendium that exists in the database are not found? Defaults to false (useful for testing).
    Should an error be return when invalid metadata is stored? Defaults to false.

The connection to the Docker API is build on dockerode which allows execution on any Docker host that exposes the port.
Most commonly, the default configuration will be used, i.e. the local Docker socket is mounted at the default location into the container running muncher (see above)


Testing is based on mocha integration tests.
A MongoDB database must be running at the default port for the tests to work and must be started manually.
Also o2r-loader and o2r-transporter must be running at their default ports.

Attention: The database is cleared completely several times during the tests!

# must start with replica set for oplog (finder) to work, see and
mongod --dbpath ./db --replSet rso2r --smallfiles;

# start o2r-loader
# start o2r-transporter

# run tests
npm test

# you can also run the tests towards a manually specified host
TEST_HOST=http://localhost:80 npm test

# stop tests after the first failing one
npm run test_bail

The archives created to upload workspaces and compendia for testing are cached.
Be aware that when you edit files in test workspaces and compendia, you must manually delete the cached files, e.g. /tmp/o2r-muncher-upload_<hash>.zip.
You can use the hash to identify tests that use the same files on Travis, as multiple tests may fail if one compendium/workspace is faulty.

To run single tests on Travis (and thereby reducing the logs of loader and muncher to only the ones of interest) you can use custom builds and overwrite only the required run command:

  - DEBUG=*,mocha:*,-modem mocha ./test/ --grep "<name of the test>"


Run container with MongoDB on host

docker run -it -p 8080:8080 -v /var/run/docker.sock:/var/run/docker.sock -e MUNCHER_MONGODB=mongodb:// -e DEBUG=muncher,muncher:* muncher

Removing all containers/images created by muncher

docker ps -a | grep erc | awk '{print $1}' | xargs --no-run-if-empty docker rm

docker images --no-trunc | grep erc | awk '{print $3}' | xargs --no-run-if-empty docker rmi -f

Steps for starting a local development environment manually

The following steps assume that you have all the required projects (o2r-contentbutler, o2r-muncher, o2r-loader, o2r-platform) in one directory. Repository updates (git pull, npm install, bower install and the like) are not shown.

mkdir /tmp/o2r-mongodb-data
mongod --dbpath /tmp/o2r-mongodb-data

# new terminal: start loader (default port 8088)

# new terminal: start muncher (default port 8080)
cd ../o2r-muncher
DEBUG=* npm start

# new terminal: run tests to add test data
npm test

# new terminal: run a webservice container in daemon mode on port 80 with (a) a proxy in front of the microservices and (b) the client project at / (must change app constant manually!)
cd ../o2r-platform
docker run --rm --name o2r-platform -p 80:80 -v $(pwd)/test/nginx.conf:/etc/nginx/nginx.conf -v $(pwd):/etc/nginx/html nginx

# do work, restart respective apps as needed

Alternatively, start the component(s) under development from your IDE(s).

Authentication and upload with curl

You can authenticate locally with OAuth via ORCID using the required configuration parameters (see project reference-implementation).

If you want to upload from the command line, make sure the account has the required level (it should by default), get the session cookie connect.sid content out of the browser and use it in the curl request:

curl --cookie connect.sid=s:S1oH7... -F "compendium=@/<path to>;type=application/zip" -F "content_type=compendium"

Create bags for testing

The following code uses to create, validate, or load and update an existing bag in place:

# create bag
python -c "import bagit; bag = bagit.make_bag('success-validate');"

# validate bag
python -c "import bagit; bag = bagit.Bag('success-load-validate'); print('Is Bag valid?', bag.validate());"

# update bag
python -c "import bagit; bag = bagit.Bag('success-load-validate');;"


o2r muncher is licensed under Apache License, Version 2.0, see file LICENSE.

Copyright (C) 2017 - o2r project.

Docker Pull Command
Source Repository