This Docker image provides an environment for grading Y86 assembly language programs in PrairieLearn. It includes necessary tools and libraries to compile, run, and evaluate Y86 code submissions. To use it, set the question's info.json to use this image as the grader environment:
{
"externalGradingOptions": {
"image": "jonatanschroeder/grader-y86:latest"
}
}
The autograder assumes that the student's submission will contain a single file with a .s or .ys extension, which is the Y86 assembly source code to be graded. The grader will compile the code and provide feedback if there are any compilation errors.
If additional content is expected to be added to the student content, such as instructor-supplied functions or constants, these should be included in the tests/suffix.s file within the question's directory. The content of this file will be appended to the student's submission before compilation. Instructors are encouraged to avoid fixed addresses in this suffix file to prevent overlapping with student code. Directives like .align can be used to set up a valid address without hardcoding specific addresses.
To run tests, the autograder will look for test files in the tests/test.js file in the question directory. This file should export an array of test cases, each containing the necessary information to execute and validate the Y86 program. The tests can check for correct output, performance metrics, or other criteria as needed.
The tests/test.js code must export an array named tests, where each element is an object representing a test case. Each test case object may contain the following properties:
name (string): The name of the test case.repeat (number, optional): The number of times to repeat the test case. This may be useful for tests that involve randomness or probabilistic outcomes. Defaults to 1 if not specified.max_points (number, optional): The number of points to assign to the test case if it passes. Defaults to 1 if not specified.description (string, optional): A description of the test case, which will be displayed to the student.message (string, optional): A starting message to display to the student. The test may append additional information to this message based on the test results.output (string, optional): A starting output to display to the student. The test may append additional information to this output based on the test results. Any errors thrown during the execution of the test will also be appended to this output.run (function): The function that implements the test case. This function will be called by the autograder to execute the test.The autograder will execute the run() function for each test case, passing in a grader object and any additional data needed for the test. The grader object provides methods to interact with the Y86 environment, such as running the compiled program and checking its output. The data parameter contains the data generated by the question's server.py file, including random parameters and correct answers.
If an error is thrown in the execution of run(), the test will be marked as failed, and the error message will be included in the output provided to the student. If the test completes without throwing an error, it will be considered passed. The run() function may return the number of points earned for the test, which will be added to the student's total score. If no value is returned, the test will be awarded the full max_points specified in the test case.
Each execution of run() starts with a fresh instance of the Y86 environment, ensuring that tests do not interfere with each other. All registers, memory, and state are reset to the values set by the student's program and any suffix code.
The grader object passed to the run() function provides the following methods:
grader.run_until_halt(): Runs the Y86 program until it reaches a halt instruction. May receive additional options for execution control:
grader.run_until_halt({ max_cycles: 100000 }): Limits execution to a maximum number of CPU instructions to prevent infinite loops. If the limit is reached, an error is thrown. By default, the limit is set to 50000 cycles.grader.run_until_halt({ breakpoints: [] }): Sets breakpoints at specified memory addresses. See the breakpoints section below for more details.grader.function_call(func_name, args): Calls a function defined in the Y86 program with the given arguments. The arguments are set to appropriate registers and the stack before calling the function, following the ISA calling conventions. The function is executed until it returns. May receive additional options:
grader.function_call(func_name, args, { max_cycles: 100000 }): Limits execution to a maximum number of CPU instructions. If the limit is reached, an error is thrown. By default, the limit is set to 50000 cycles.grader.function_call(func_name, args, { breakpoints: [] }): Sets breakpoints at specified memory addresses. See the breakpoints section below for more details.grader.function_call(func_name, args, { check_stack_pointer: false }): By default, the stack pointer is initialized to a large address before the function call and checked after the function returns to ensure it has not been modified. Setting this option to false disables this check. This may be used in tests that need to fine-tune the stack pointer behavior for specific scenarios.grader.function_call(func_name, args, { check_callee_save: false }): By default, the callee-saved registers (%rbx, %rbp, etc.) are initialized with random values and checked after the function returns to ensure they have not been modified. Setting this option to false disables this check.grader.function_call(func_name, args, { allow_halt: true }): By default, if the function being called executes a halt instruction, an error is thrown. Setting this option to true allows the function to execute a halt instruction without throwing an error. In that case, the execution will be interrupted by the halt instruction.grader.set_register_value(register, value): Sets the specified register to the given value. The register parameter may be a string representing the register name (e.g., %rax, %rbx, etc.) or the numeric representation of the register (e.g., 0 for %rax or 1 for %rcx). The value parameter should be a number representing the value to set, either as a number or a bigint. The value will be truncated to fit the register size (typically 64 bits).grader.get_register_value(register): Retrieves the current value of the specified register. The register parameter may be a string representing the register name (e.g., %rax, %rbx, etc.) or the numeric representation of the register (e.g., 0 for %rax or 1 for %rcx). The method returns the value of the register as a bigint.grader.write_memory(address, value, size = 8): Sets the memory at the specified address to the given value. The address parameter should either be a number representing the memory address to write to, or a string representing a label set in the program. If the value parameter is a number or bigint, it will be written directly to memory at the specified address, with the endianness associated to the architecture (e.g., little-endian for Y86). If the value parameter is an array, the values will be written sequentially starting at the specified address. The optional size parameter specifies the number of bytes to write (default is 8 bytes). For array values, the size parameter specifies the size of each individual element in the array.grader.read_memory(address, size = 8): Reads the memory at the specified address and returns its value. The address parameter should either be a number representing the memory address to read from, or a string representing a label set in the program. The optional size parameter specifies the number of bytes to read (default is 8 bytes). The method returns the value read from memory as a bigint, interpreted with the endianness associated to the architecture (e.g., little-endian for Y86).grader.label_to_memory_address(label): Converts a label defined in the Y86 program to its corresponding memory address. The label parameter should be a string representing the label name. The method returns the memory address associated with the label as a number. If the label does not exist in the student code or the suffix, an error is thrown.grader.scramble_caller_save_registers(): Randomizes the values of the caller-saved registers (e.g., %rax, %rcx, etc.) to help detect unintended dependencies on register values across function calls. This method is typically called before invoking a function, or in a breakpoint inside a grader-provided function, to ensure that the student function does not rely on specific values in these registers.grader.assert_register_value(register, expected_value): Asserts that the specified register has the expected value. The register parameter may be a string representing the register name (e.g., %rax, %rbx, etc.) or the numeric representation of the register (e.g., 0 for %rax or 1 for %rcx). The expected_value parameter should be a number or bigint representing the expected value of the register. If the actual value does not match the expected value, an error is thrown with an appropriate message. The comparison is limited to the size of the register being compared (e.g., 64 bits for general purpose y86 registers). May receive additional parameters:
grader.assert_register_value(register, expected_value, {comparison = "=="}): specifies the type of comparison to perform (default is "=="). Pre-defined comparison options include "==", "!=", "<", "<=", ">", and ">=". It may also be assigned to a function that takes two bigint parameters (actual and expected) and returns a boolean indicating whether the assertion passes.grader.assert_memory_value(address, expected_value): Asserts that the memory at the specified address has the expected value. The address parameter should either be a number representing the memory address to read from, or a string representing a label set in the program. The expected_value parameter may be a number or bigint representing the expected value of the memory location, or an array of numbers or bigints; if it is an array, the method will read sequential memory locations starting at the specified address and compare each value in the array to the corresponding memory location. If the actual value does not match the expected value, an error is thrown with an appropriate message. May receive additional parameters:
grader.assert_memory_value(address, expected_value, {size = 8}): specifies the number of bytes to read (default is 8 bytes). If the expected_value parameter is an array, the size parameter specifies the size of each individual element in the array.grader.assert_memory_value(address, expected_value, {comparison = "=="}): specifies the type of comparison to perform (default is "=="). Pre-defined comparison options include "==", "!=", "<", "<=", ">", and ">=". It may also be assigned to a function that takes two bigint parameters (actual and expected) and returns a boolean indicating whether the assertion passes. The comparison is limited to the specified size of the memory location being compared.grader.assert_memory_value(address, expected_value, {address_label = null}): provides a label name for the address being checked, which will be included in the error message if the assertion fails. If not specified, a representation of the address itself will be used in the error message.grader.assert_non_overlapping_labels(labels): Asserts that the specified labels do not overlap in memory. The labels parameter should be an array representing the label names to be verified. If any pair of labels overlap in memory, an error is thrown with an appropriate message indicating which labels are overlapping. Each label is represented as either a string, corresponding to a label name, or an object containing the label name and size, in bytes, of the memory region to check for overlap. If the size is not specified, it defaults to 8 bytes.Breakpoints may be set during program execution using the breakpoints option in the grader.run_until_halt() and grader.function_call() methods. Breakpoints allow the grader to pause execution at specific memory addresses, enabling inspection of the program state at those points.
When a breakpoint is hit, execution is paused before executing the instruction at the breakpoint address. The grader can then inspect register values, memory contents, and other state information. After inspection, execution can be resumed until the next breakpoint or until the program halts.
The breakpoints argument corresponds to an array of objects. Each object will have two properties:
pos corresponds to the address in memory where the breakpoint will be triggered. This can be a numeric address (either a number or bigint), or a string representing a label in the code. The latter is often useful to set breakpoints at the start of individual functions.check() is a callback function to be called when the breakpoint is hit. It does not receive any arguments.Inside the check() function, you can use the grader's assertion methods to verify the program state at the breakpoint.
Here is an example of a tests/test.js file that tests a Y86 program implementing a recursive factorial function, and a program that calls this function and stores the result in memory:
export const tests = [
{
name: "Factorial of 7",
max_points: 2,
description:
"Checks if the factorial function returns the correct result for input 7.",
run: async (grader) => {
// Call the factorial function
await grader.function_call("factorial", [7]);
// Assert that the return value in %rax is 5040
grader.assert_register_value("%rax", 5040);
},
},
{
name: "Factorial is recursive",
max_points: 3,
description:
"Ensures that the factorial function uses recursion by checking for stack usage.",
run: async (grader) => {
let next_recursive_argument = 4;
grader.function_call("factorial", [5], {
breakpoints: [
{
pos: "factorial",
check: () => {
grader.assert_register_value("%rdi", next_recursive_argument);
if (next_recursive_argument <= 0) {
throw new Error("Factorial called beyond base case");
}
next_recursive_argument--;
},
},
],
});
if (next_recursive_argument > 1) {
throw new Error(
"factorial was not called recursively or did not reach base case",
);
}
grader.assert_register_value("%rax", 120);
},
},
{
name: "Program saves factorial result to memory",
max_points: 2,
description:
"Checks if the program correctly retrieves the input from the 'input' label and saves the factorial of the input to the 'result' label.",
run: async (grader) => {
// Ensure that 'input' and 'result' labels exist and are not in the same location
await grader.assert_non_overlapping_labels(["input", "result"]);
// Set up the input value in memory
grader.write_memory("input", 6);
// Run the program until halt
await grader.run_until_halt();
// Assert that the result in memory is correct
await grader.assert_memory_value("result", 720);
},
},
];
Here is an example of a question that asks a user to store a value in memory that was randomly generated by the server.py file and stored in data["params"]["random_value"]:
export const tests = [
{
name: "Store random value in memory",
description:
"Checks if the program correctly stores the randomly generated value in the 'output' label.",
run: async (grader, data) => {
// Run the program until halt
await grader.run_until_halt();
// Assert that the value in memory matches the randomly generated value
await grader.assert_memory_value("output", data.params.random_value);
},
},
];
Content type
Image
Digest
sha256:ebcf8287f…
Size
55.5 MB
Last updated
about 1 month ago
docker pull jonatanschroeder/grader-y86