Public | Automated Build

Last pushed: 9 months ago
Short Description
A helper container to execute a program and store results.
Full Description


A helper container to execute a program and store results.

How to use

1. Build a container including your code.

You need to build a container based on Roadie.
In your container, your codes should be put in /data.
You can choose Google Cloud Storage or MongoDB as a place to store outputs.

Here is a sample Dockerfile assuming your code is and you choose sample-bucket in Google Cloud Storage as the storage.

FROM jkawamoto/roadie

# Import your code.
ADD /data

# Set configurations.
CMD ["", "gcs", "sample-bucket"]

Then, you can build the container named yourname/main.

$ docker build -t yourname/main .

2. Run the container.

Assuming the name of your container is yourname/main, you can run the container by the following command.

$ docker run -d yourname/main

Then, the executed result will be stored in a place specified in the Dockerfile.
In the above example, a file stdout consisting of outputs written in stdout will be stored in sample-bucket.
On the other hand, stderr will be connected to stdout of the container so that you can watch it via such as docker logs.
If your program produces other output files, see detailed information below.

Detailed information

Containers delivered from roadie will execute a helper script at a run time.
Then, this script invoke your program. Actually, the script will invoke the first argument.
For example, you can run simple echo command by the following command.

$ docker run -d junpei/roadie "echo 'Hello world!'" gce "sample-bucket"

The format of the script is as follows.

usage: [-h] [--observe [OBSERVE [OBSERVE ...]]] [--cwd CWD]
                     [--shutdown] [--zip] [--log LOG] [--stderr STDERR]
                     cmd {gcs,mongo,local} ...

positional arguments:
  cmd                   Command line to be run.
    gcs                 Storing outputs into Google Cloud Storage.
    mongo               Storing outputs into MongoDB.
    local               Storing outputs into local filesystem.

optional arguments:
  -h, --help            show this help message and exit
  --observe [OBSERVE [OBSERVE ...]]
                        glob patterns of files to be stored.
  --cwd CWD             Change working directory (default: /data).
  --shutdown            Shutdown after the program ends (working only in
                        Google Compute Engine)
  --zip                 Files specified in overve option will be zipped.
  --log LOG             Config file of loggers.
  --stderr STDERR       Specify where stderr should be stored (default:

If your program will produce some files, specify file-name patterns at --observe argument to be stored.
UNIX based patterns such as result-*.out, etc. are supported.
cmd should be quoted if it has spaces.

Store to Google Cloud Storage

To store outputs to GCS, use gcs option.

usage: cmd gcs [-h] [--prefix PREFIX] bucket

positional arguments:
  bucket           Bucket name.

optional arguments:
  -h, --help       show this help message and exit
  --prefix PREFIX  Prefix of stored files.

Store to MongoDB

To store outputs to MongoDB, use mongo option.

usage: cmd mongo [-h] [--host HOST] [--port PORT] db collection

positional arguments:
  db           Database name.
  collection   Collection name.

optional arguments:
  -h, --help   show this help message and exit
  --host HOST  Host name of MongoDB server.
  --port PORT  Port number of MongoDB server.

Store to local filesystem

To store outputs to local, use local option.

usage: cmd local [-h] dir

positional arguments:
  dir         Directory for storing output.

optional arguments:
  -h, --help  show this help message and exit


This software is released under the MIT License, see LICENSE.

Docker Pull Command
Source Repository