6. Automated program assessment¶
6.1. Introduction¶
This chapter is intended for teachers that want to write program tests. VPL supports automated program assessment using the Basic Input/Output Test Evaluation System (BIOTES).
BIOTES is provided by VPL out of the box. This chapter shows the test case description language used by BIOTES. The teachers, using this language, write the test cases in the vpl_evaluate.cases file (action menu “Test cases”) for evaluating the students’ programs.
The language uses statements with the format “statement_name = value”. Based on the statement type the value can take only one line or spans multiple ones. A multiline value ends when another statement appears. Notice that this behavior limits the valid contents of the values of the statement. The statement name is case insensitive. A basic test case definition includes a case name, the input we want to provide to the student’s program, and the output we expect from it. We can also configure other stuff, as the penalization for failed tests, timeout, etc. VPL will run the evaluation using the test cases in vpl_evaluate.cases and generating a report of failed cases and the mark obtained.
6.2. Basic testing¶
6.2.1. Case¶
This statement starts a new case definition and states the case description.
Format: "Case = Test case description"
The case description occupies only one line. This description will appear in the report if the case fails.
Example:
6.2.2. Input¶
This statement defines the text to send to the student program as input. Each case requires one and only one input statement. Its value can span multiple lines. The system does not control if the student program reads or not the input.
Format: "Input = text"
Example 1:
Example 2:
Example 3:
Example 4:
6.2.3. Output¶
The output statement defines a possible correct output of the student program for the input of the current case. A test case can have multiple output statements and at least one. If the program output matches one of the output statements the test case succeeds, else fails. There are four kinds of values for an output statement: numbers, text, exact text and regular expression.
Format: "Output = value"
The value of the output can span multiple lines.
6.2.3.1. Checking only numbers¶
To define this type of output check, you must use only numbers as values in the output statement. The values can be integers or floating numbers. This type of output checks numbers in the output of the student’s program, ignoring the rest of the text. The output of the student’s program is filtered removing the non-numeric text. Finally, the system compares the resulting numbers of the output with the ones expected for the case, using a tolerance when comparing floating numbers.
Example 1:
Example 2:
6.2.3.2. Checking text¶
This type of output is a nonstrict text check comparing only words in the output of the student’s program, the comparison is case-insensitive and ignoring the punctuation marks, spaces, tabs, and newlines. To define this type of output check, you must use text with not only numbers and not starting with a slash nor being inside double-quotes. A filter removes punctuation marks, spaces, tabs, and newlines from the output of the student’s program, leaving a separator between each word. Numbers are not punctuation marks and so they are not removed. Finally, the system compares case-insensitive the resulting text with the expected output.
Example:
6.2.3.3. Checking exact text¶
This type of output checks the exact text on the output from the student’s program. To define this type of output check, the teacher must use text enclosed in double quotes. The system compares the output of the program with the defined output (removing double quotes).
Example 1:
Example 2:
6.2.4. Checking regular expression¶
The evaluator can define this type of check starting the output value with a slash “/” and ending with another slash “/” plus optionally one or several modifiers. This format is similar to JavaScript literal regular expression, but using POSIX regex instead.
Example:
Student’s program output that matches the output definition
6.2.5. Multiple output checking¶
The test case definition may contain multiple output statements meaning that if any of them matches the case succeeds.
Example 1:
Student’s program output that matches the output definition
Student’s program output that does not matches the output definition
Example 2:
6.2.6. Penalizations and final grade¶
A test case fails if its output does not match with an expected value. By default, the penalty applied when a test case fails is the “grade_range/number_of_cases”. The penalties of all failed test cases are sum to obtain the overall penalization. The final grade is the maximum mark less the total penalization. The final grade value never is less than the minimum grade or greater than the maximum grade of the VPL activity.
6.3. Advanced testing¶
6.3.1. Custom penalization¶
The evaluator must use the following statement if wants to customize the penalty of a test case:
Format: "Grade reduction = [ value | percent% ]"
The penalty can be a percentage or a specific value. The final grade will be maximum grade for the activity minus the overall penalization. If the result value is less than the minimum grade, the minimum is applied.
Example:
6.3.2. Controlling the messages in the output report¶
When a test fails, BIOTES adds to the report the details of the input, the output expected and the output found. When a test case fails and has a “Fail message” statement, the system instead of showing the default input/output report, shows that message.
The “Fail message” statement allows the evaluator to hide the data used in the case. A student knowing the inputs and each output expected might code a solution that passes the tests without resolving the problem. If the fail message statement appears in a test case, and the case fails, the report will only contain the message in this statement.
Format: "Fail message = message"
This statement must go before input.
Example:
6.3.3. Running another program¶
The teacher can use another program to test a different feature of the student’s program. Among other possibilities, this allows running static/dynamic analysis of the student code, e.g., the teacher can run checkstyle [1] to check the style of the student’s java code. The “Program to run” replaces the student’s program for another one in the test case that uses it.
Format: "Program to run = path"
Example:
Note
If you plan to use a custom script as in the example remember to mark it in the Files to keep when running.
6.3.4. Program arguments¶
This statement allows sending information as command-line arguments to the student program or the “program to run” if set. Notice that this statement is compatible with the input statement.
Format: "Program arguments = arg1 arg2 …"
Example 1:
This example shows how to use the “Program to run” and “Program arguments” statements to check if the student’s program creates a file with a name passed as a command-line argument.
Example 2:
The following example shows how the teacher can use the “Program to run” and “Program arguments” statements to evaluate a SQL query exercise using different data sets.
Note
If you plan to use a custom script as in the example remember to mark it in the Files to keep when running.
6.3.5. Expected exit code¶
This statement sets the expected exit code of the program case execution. The test case succeeds if the exit code matches. Notice that the test case also succeeds if an output matches.
Format: "Expected exit code = number"
The next example that shows the possibilities of “Program to run” and “Program arguments” statements to execute different programs. The first case changes the name of a file, the second compiles the file, and the third run the resulting program.
6.3.6. Variation¶
This statement set that the test case must only apply if the indicated variation was assigned to the current student. If the variation does not match then the test case is fully ignored.
Format: "Variation = identification"
[1] | http://checkstyle.sourceforge.net/ |
For more details about VPL, visit the VPL home page or the VPL plugin page at Moodle.