A docker container that includes a stable Debian Linux and convenience commands for grading a student submission.
Commands include tools from capturing the test output to submitting a feedback back to the grading service.
This container is just a base and does not include tools for different programming and scripting languages.
Thus, actual grading/assessment containers should derive from this container via the FROM instruction in their Dockerfiles.
You can find many grading containers on the apluslms page on docker hub.
Containers starting with
grade- are relevant for this purpose.
Version of grading base has format
Major is updated when there is backwards incompatible change.
For example, when deprecated tool is removed.
Minor number is updated every time there is something new, where that bug fixes, added features or update on the debian base image.
Derived containers typically use format
<tool version>-<garding-base version>u<update>,
where tool version is the version from upstream, grading-base is the version of this container and update is number that indicates update on containerisation.
So, for example, grade-python
3.5-2.2u1 means that there is python 3.5 on top of grading-base 2.2 and it's first update after initial release.
Update part is optional and can be omitted.
The container entry point, the grade wrapper
Container has grade wrapper
/gw set as the entry point.
This means that everything executed within this container (or derivative) is executed through this script.
The wrapper will take care of redirecting output (stdout and stderr) to
In addition, the script will make sure that the working directory is set correctly.
After the grading script has been executed, the wrapper will execute
Thus, the grading script does not need to call
The wrapper has three different operations, based on the first argument.
If the first argument starts with
then arguments are executed and the wrapper itself stops here.
No redirections are done and
gradeis not executed.
If the first argument is
--sh, then rest of the arguments are evaluated with
sh -c "$*".
This makes strings with quoted text possible in mooc-grader environment.
The typical case. Argument list is executed as is.
The exit code from the grading script is stored in the grading-script-errors, but the grade wrapper always exits with 0.
Files for the solution submitted by the student are stored here.
This path can be used as working directory.
Exercise files are mounted here in a read only mode.
With mooc-grader, that means the path you entered in
Typically, this includes
run.shand unit test files.
Grading script should store feedback in this path.
Most utility commands do so, including
In the future, mooc-grader will read files from this path instead of expecting a post request.
Valid format for the data is
You can use
pointsto update this file.
Points are read by
Feedback that is going be presented in the LMS.
Typically, data is in html format.
You should use
preto wrap plain text to html.
You can capture feedback with
capture pre <command>, for example.
Error messages from a student or a testing code.
gradewill append this inside a pre tag at the end of the out file.
This file typically contains stderr from the test code.
captureredirects stderr to this file.
If stderr is normal output (e.g. with python unit tests), then you should consider using
Errors that should be visible only to the staff.
This file is added as a part of
errorsand send to the grading service.
Then, data can be shown in the LMS to the course staff.
Data is expected to be plain text messages from
run.shand other tools.
/gwredirects stdout and stderr from the
run.shto this file.
If this file contains
true, then the submission is set in an error state in the LMS.
You can use
set-errorin your script to set this file.
A single feedback item.
A set of these directories is considered the feedback list.
Typically, one folder matches one test case.
For now, this feedback list is not fully supported and the tool
create-test-feedbackis provided to render these directories to
Nevertheless, this allows building tools and scripts that export feedback in this format.
In future, different templates can be used to render the feedback list in the grading service.
gradetool will automatically merge these directories to a single html file, if grading service doesn't support feedback as a list.
For every numbered path (feedback item), there can exists any number of files, but following files are typical.
Following files are supported by
create-test-feedbackand a basic template on the grading service.
More advanced or specific templates can of course deviate from those.
.../title: A title or a display name for this feedback item.
.../status: The status of this feedback. Valid values:
.../points: Points given from this item.
.../max_points: Maximum points that could be given from this item.
.../out.txt: Stdout from the test.
.../err.txt: Stderr from the test.
Following utility commands are provided in the path.
testcase [-t title] [-p points_on_success] [-s only_if_zero] CMD...
Defines a single test case.
Useful when running multiple commands for grading.
/feedback/<number>directory with increasing number for every call.
titleis provided, it's stored in
points_on_successis provided, it will be written to
If CMD exits with 0, then it will also be written to
only_if_zerois provided and not 0, then CMD is not executed and
testcaseexits with non-zero exit code.
Status of this test is stored in
capture [-o out] [-e err] [-u user] CMD...
Changes the ownership of the working directory to
executes the command as
and captures the stdout and stderr to
In addition to out and err redirection,
capturewill make sure that the environment does not contain
SID(the variables used for posting feedback).
capture pre ./my_testwill execute
nobodyand redirect stdout and stderr to
If you need to run captured code as root, then use
pre [-c class] CMD...
Wraps the stdout of the command into <pre> HTML element
and escapes any HTML inside. Can be chained, e.g.
capture pre ./my_assessment_code.sh arg1 arg2
-ccan be used to set HTML class for the element.
points <points> [only_if_zero]
pointsto given and maximum points in
If the file exists, it will be updated.
only_if_zerois given and not zero, then only maximum points are changed.
This can be used in conjunction with
captureto give points conditionally.
capture ./my_assessment_code.sh points 10 $?
$?is the exit code from
title [-c class] [-e element] [-O] <text>
textinside HTML element
The element is
h1by default, but can be changed with
-ccan be used to set HTML class for the element.
-Ochanges the output to stdout.
Submits the feedback back to the grading service.
The grade wrapper will execute this command after the grading script.
This command exits without changes if it is executed for the second time,
thus grading script can call this too, but is not required to.
If no argument is provided, there isn't
/feedback/pointsand any of
create-test-feedbackis executed before reading the feedback.
Gets granted points in priority order from:
a) argument in format "5/10"
b) /feedback/points file in format "5/10"
c) /feedback/out including lines "TotalPoints: 5" and "MaxPoints: 10"
Gets feedback as HTML from /feedback/out,
and wraps /feedback/err into pre with
alert-dangerclass at the end.
If /feedback/appendix exists it will be included at the very end.
stdio-diff [-c] [-s] [-S] [-p pass_text] [-P] [-f fail_text] <in_file> <expected_out> CMD...
Simple text comparison wrapper based on
expected_out, script will execute CMD with
in_fileas stdin and store it's stdout.
Finally, it will compare
expected_outto stdout of the CMD.
The output from diff is revealed to the user, unless
-cmakes output case sensitive.
-smakes output space sensitive. There needs to be space between words, but amount doesn't matter.
-Smakes diff totally ignore space (removes all space before comparison).
okis printed when the output matches the expected.
The string can be changed to a custom value with
-por to nothing with
This command can be used with
capture pre stdio-diff in.txt out.txt ./student_code.
Combines feedback items from
This is automatically executed by
gradeif preconditions apply.
Useful for unit test frameworks that produce the feedback into stderr.
Converts recursively all files in working directory from any recognizable
charsets into desired charset, e.g. utf-8. Discards unconvertible characters.
May be useful to sanitize output from exotic student environments.
Sets submission error state by writing
msgis provided, then it is written as the reason to
These utilities are used internally, but can also benefit when writing grading scripts.
First one provides the other three.
Read linked documentations for more details.
gitlab-api-query.py GIT_HOST [api_token] [require_fork]
Reads user repository URL from submitted
e.g. written by mooc-grader types.stdasync.acceptGitAddress,
and then checks information in GitLab API at given host.
Prints message and exits non zero if repository is public.
To access GitLab API as privileged user. Can also check
that named private repository exists and the fork argument.
Requires the submitted repository to be a fork from the
named one, e.g. lehtint6/material-delivery.
git-clone-submission PRIVATE_KEY_PATH [file paths...]
Clones repository from URL submitted as
into /submission/user-repo. The private ssh key allows
access to repositories where the public key pair is added
as a deploy key. Prints status and exits non zero if fails.
List of expected files to check inside the repository.
The files are moved into /submission/user and the
contents are written as appendix to end of feedback.