6. Automated program assessment¶
6.1. Introduction¶
This chapter is intended for teachers that want to write program tests. VPL supports automated program assessment using the Basic Input/Output Test Evaluation System (BIOTES).
BIOTES is provided by VPL out of the box. This chapter shows the test case description language used by BIOTES. The teachers, using this language, write the test cases in the vpl_evaluate.cases file (action menu “Test cases”) for evaluating the students’ programs.
The language uses statements with the format “statement_name = value”. Based on the statement type the value can take only one line or spans multiple ones. A multiline value ends when another statement appears. Notice that this behavior limits the valid contents of the values of the statement. The statement name is case insensitive. A basic test case definition includes a case name, the input we want to provide to the student’s program, and the output we expect from it. We can also configure other stuff, as the penalization for failed tests, timeout, etc. VPL will run the evaluation using the test cases in vpl_evaluate.cases and generating a report of failed cases and the mark obtained.
6.2. Basic testing¶
6.2.1. Case¶
This statement starts a new case definition and states the case description.
Format: "Case = Test case description"
The case description occupies only one line. This description will appear in the report if the case fails.
Example:
Minimum test cases definition
Evaluation report showing the failed test case
Evaluation report showing the passed test case
6.2.2. Input¶
This statement defines the text to send to the student program as input. Each case requires one and only one input statement. Its value can span multiple lines. The system does not control if the student program reads or not the input.
Format: "Input = text"
Example 1:
Example of input of a line of numbers
Example 2:
Example of input of a multiple lines of numbers
Example 3:
Example of input of a line of text
Example 4:
Example of input of a multiple lines of text
6.2.3. Output¶
The output statement defines a possible correct output of the student program for the input of the current case. A test case can have multiple output statements and at least one. If the program output matches one of the output statements the test case succeeds, else fails. There are four kinds of values for an output statement: numbers, text, exact text and regular expression.
Format: "Output = value"
The value of the output can span multiple lines.
6.2.3.1. Checking only numbers¶
To define this type of output check, you must use only numbers as values in the output statement. The values can be integers or floating numbers. This type of output checks numbers in the output of the student’s program, ignoring the rest of the text. The output of the student’s program is filtered removing the non-numeric text. Finally, the system compares the resulting numbers of the output with the ones expected for the case, using a tolerance when comparing floating numbers.
Example 1:
Example of output of type number
Example of incorrect program for output of type number
Example of correct program for output of type number
Example 2:
Example of output of float numbers
Example of incorrect program for output of float numbers
Example of correct program for output of float numbers
6.2.3.2. Checking text¶
This type of output is a nonstrict text check comparing only words in the output of the student’s program, the comparison is case-insensitive and ignoring the punctuation marks, spaces, tabs, and newlines. To define this type of output check, you must use text with not only numbers and not starting with a slash nor being inside double-quotes. A filter removes punctuation marks, spaces, tabs, and newlines from the output of the student’s program, leaving a separator between each word. Numbers are not punctuation marks and so they are not removed. Finally, the system compares case-insensitive the resulting text with the expected output.
Example:
Example of output of type text
Student’s program that matches the defined output
Student’s program that pass the test
6.2.3.3. Checking exact text¶
This type of output checks the exact text on the output from the student’s program. To define this type of output check, the teacher must use text enclosed in double quotes. The system compares the output of the program with the defined output (removing double quotes).
Example 1:
Example of output of type exact text
Student’s program output that matches the defined output
Example 2:
Example of output of type exact text
Student’s program output that matches the defined output
6.2.4. Checking regular expression¶
The evaluator can define this type of check starting the output value with a slash “/” and ending with another slash “/” plus optionally one or several modifiers. This format is similar to JavaScript literal regular expression, but using POSIX regex instead.
Example:
Example of output of type regular expression
Student’s program output that matches the output definition
6.2.5. Multiple output checking¶
The test case definition may contain multiple output statements meaning that if any of them matches the case succeeds.
Example 1:
Example of multiple outputs of different types
Student’s program output that matches the output definition
Student’s program output that does not matches the output definition
Example 2:
Example of multiple outputs of numbers type
6.2.6. Penalizations and final grade¶
A test case fails if its output does not match with an expected value. By default, the penalty applied when a test case fails is the “grade_range/number_of_cases”. The penalties of all failed test cases are sum to obtain the overall penalization. The final grade is the maximum mark less the total penalization. The final grade value never is less than the minimum grade or greater than the maximum grade of the VPL activity.
6.3. Advanced testing¶
6.3.1. Custom penalization¶
The evaluator must use the following statement if wants to customize the penalty of a test case:
Format: "Grade reduction = [ value | percent% ]"
The penalty can be a percentage or a specific value. The final grade will be maximum grade for the activity minus the overall penalization. If the result value is less than the minimum grade, the minimum is applied.
Example:
Example of fail penalization customized
Student’s program output that fails the evaluation
6.3.2. Controlling the messages in the output report¶
When a test fails, BIOTES adds to the report the details of the input, the output expected and the output found. When a test case fails and has a “Fail message” statement, the system instead of showing the default input/output report, shows that message.
The “Fail message” statement allows the evaluator to hide the data used in the case. A student knowing the inputs and each output expected might code a solution that passes the tests without resolving the problem. If the fail message statement appears in a test case, and the case fails, the report will only contain the message in this statement.
Format: "Fail message = message"
This statement must go before input.
Example:
Example of fail message cuztomization
Student’s program output that fails the evaluation
6.3.3. Running another program¶
The teacher can use another program to test a different feature of the student’s program. Among other possibilities, this allows running static/dynamic analysis of the student code, e.g., the teacher can run checkstyle [1] to check the style of the student’s java code. The “Program to run” replaces the student’s program for another one in the test case that uses it.
Format: "Program to run = path"
Example:
Example of using “Program to run” statement
Note
If you plan to use a custom script as in the example remember to mark it in the Files to keep when running.
6.3.4. Program arguments¶
This statement allows sending information as command-line arguments to the student program or the “program to run” if set. Notice that this statement is compatible with the input statement.
Format: "Program arguments = arg1 arg2 …"
Example 1:
This example shows how to use the “Program to run” and “Program arguments” statements to check if the student’s program creates a file with a name passed as a command-line argument.
Example of using “Program arguments” statement and “Program arguments” statements
Code of the check_file_exist.sh script
Example 2:
The following example shows how the teacher can use the “Program to run” and “Program arguments” statements to evaluate a SQL query exercise using different data sets.
Example of using “Program to run” and “Program arguments” statements
Note
If you plan to use a custom script as in the example remember to mark it in the Files to keep when running.
6.3.5. Expected exit code¶
This statement sets the expected exit code of the program case execution. The test case succeeds if the exit code matches. Notice that the test case also succeeds if an output matches.
Format: "Expected exit code = number"
The next example that shows the possibilities of “Program to run” and “Program arguments” statements to execute different programs. The first case changes the name of a file, the second compiles the file, and the third run the resulting program.
Example of using “Program to run”, “Program arguments”, “Expected exit code” statements
6.3.6. Variation¶
This statement set that the test case must only apply if the indicated variation was assigned to the current student. If the variation does not match then the test case is fully ignored.
Format: "Variation = identification"
Example of using the “Variation” statement
| [1] | http://checkstyle.sourceforge.net/ |
For more details about VPL, visit the VPL home page or the VPL plugin page at Moodle.