6. Automated program assessment

6.1. Introduction

This chapter is intended for teachers that want to write program tests. VPL supports automated program assessment using the Basic Input/Output Test Evaluation System (BIOTES).

BIOTES is provided by VPL out of the box. This chapter shows the test case description language used by BIOTES. The teachers, using this language, write the test cases in the vpl_evaluate.cases file (action menu “Test cases”) for evaluating the students’ programs.

The language uses statements with the format “statement_name = value”. Based on the statement type the value can take only one line or spans multiple ones. A multiline value ends when another statement appears. Notice that this behavior limits the valid contents of the values of the statement. The statement name is case insensitive. A basic test case definition includes a case name, the input we want to provide to the student’s program, and the output we expect from it. We can also configure other stuff, as the penalization for failed tests, timeout, etc. VPL will run the evaluation using the test cases in vpl_evaluate.cases and generating a report of failed cases and the mark obtained.

6.2. Basic testing

6.2.1. Case

This statement starts a new case definition and states the case description.

Format: "Case = Test case description"

The case description occupies only one line. This description will appear in the report if the case fails.

Example:

_images/basic_case_definition.png

Minimum test cases definition

_images/basic_case_fail.png

Evaluation report showing the failed test case

_images/basic_case_pass.png

Evaluation report showing the passed test case

6.2.2. Input

This statement defines the text to send to the student program as input. Each case requires one and only one input statement. Its value can span multiple lines. The system does not control if the student program reads or not the input.

Format: "Input = text"

Example 1:

_images/basic_input_definition1.png

Example of input of a line of numbers

Example 2:

_images/basic_input_definition2.png

Example of input of a multiple lines of numbers

Example 3:

_images/basic_input_definition3.png

Example of input of a line of text

Example 4:

_images/basic_input_definition4.png

Example of input of a multiple lines of text

6.2.3. Output

The output statement defines a possible correct output of the student program for the input of the current case. A test case can have multiple output statements and at least one. If the program output matches one of the output statements the test case succeeds, else fails. There are four kinds of values for an output statement: numbers, text, exact text and regular expression.

Format: "Output = value"

The value of the output can span multiple lines.

6.2.3.1. Checking only numbers

To define this type of output check, you must use only numbers as values in the output statement. The values can be integers or floating numbers. This type of output checks numbers in the output of the student’s program, ignoring the rest of the text. The output of the student’s program is filtered removing the non-numeric text. Finally, the system compares the resulting numbers of the output with the ones expected for the case, using a tolerance when comparing floating numbers.

Example 1:

_images/basic_input_definition1.png

Example of output of type number

_images/basic_input_definition1_fail.png

Example of incorrect program for output of type number

_images/basic_input_definition1_pass.png

Example of correct program for output of type number

Example 2:

_images/basic_output_numbers2.png

Example of output of float numbers

_images/basic_output_numbers2_fail.png

Example of incorrect program for output of float numbers

_images/basic_output_numbers2_pass.png

Example of correct program for output of float numbers

6.2.3.2. Checking text

This type of output is a nonstrict text check comparing only words in the output of the student’s program, the comparison is case-insensitive and ignoring the punctuation marks, spaces, tabs, and newlines. To define this type of output check, you must use text with not only numbers and not starting with a slash nor being inside double-quotes. A filter removes punctuation marks, spaces, tabs, and newlines from the output of the student’s program, leaving a separator between each word. Numbers are not punctuation marks and so they are not removed. Finally, the system compares case-insensitive the resulting text with the expected output.

Example:

_images/basic_output_text1.png

Example of output of type text

_images/basic_output_text1_program.png

Student’s program that matches the defined output

_images/basic_output_text1_pass.png

Student’s program that pass the test

6.2.3.3. Checking exact text

This type of output checks the exact text on the output from the student’s program. To define this type of output check, the teacher must use text enclosed in double quotes. The system compares the output of the program with the defined output (removing double quotes).

Example 1:

_images/basic_output_exactext1.png

Example of output of type exact text

_images/basic_output_exactext1_pass.png

Student’s program output that matches the defined output

Example 2:

_images/basic_output_exactext2.png

Example of output of type exact text

_images/basic_output_exactext2_pass.png

Student’s program output that matches the defined output

6.2.4. Checking regular expression

The evaluator can define this type of check starting the output value with a slash “/” and ending with another slash “/” plus optionally one or several modifiers. This format is similar to JavaScript literal regular expression, but using POSIX regex instead.

Example:

_images/basic_output_regex1.png

Example of output of type regular expression

_images/basic_output_regex1_pass1.png _images/basic_output_regex1_pass2.png

Student’s program output that matches the output definition

6.2.5. Multiple output checking

The test case definition may contain multiple output statements meaning that if any of them matches the case succeeds.

Example 1:

_images/basic_multioutput1.png

Example of multiple outputs of different types

_images/basic_multioutput1_pass1.png _images/basic_multioutput1_pass2.png _images/basic_multioutput1_pass3.png

Student’s program output that matches the output definition

_images/basic_multioutput1_fail1.png _images/basic_multioutput1_fail2.png

Student’s program output that does not matches the output definition

Example 2:

_images/basic_multioutput2.png

Example of multiple outputs of numbers type

6.2.6. Penalizations and final grade

A test case fails if its output does not match with an expected value. By default, the penalty applied when a test case fails is the “grade_range/number_of_cases”. The penalties of all failed test cases are sum to obtain the overall penalization. The final grade is the maximum mark less the total penalization. The final grade value never is less than the minimum grade or greater than the maximum grade of the VPL activity.

6.3. Advanced testing

6.3.1. Custom penalization

The evaluator must use the following statement if wants to customize the penalty of a test case:

Format: "Grade reduction = [ value | percent% ]"

The penalty can be a percentage or a specific value. The final grade will be maximum grade for the activity minus the overall penalization. If the result value is less than the minimum grade, the minimum is applied.

Example:

_images/advanced_grade_reduction1.png

Example of fail penalization customized

_images/advanced_grade_reduction1_fail.png

Student’s program output that fails the evaluation

6.3.2. Controlling the messages in the output report

When a test fails, BIOTES adds to the report the details of the input, the output expected and the output found. When a test case fails and has a “Fail message” statement, the system instead of showing the default input/output report, shows that message.

The “Fail message” statement allows the evaluator to hide the data used in the case. A student knowing the inputs and each output expected might code a solution that passes the tests without resolving the problem. If the fail message statement appears in a test case, and the case fails, the report will only contain the message in this statement.

Format: "Fail message = message"

This statement must go before input.

Example:

_images/advanced_fail_message1.png

Example of fail message cuztomization

_images/advanced_fail_message1_fail.png

Student’s program output that fails the evaluation

6.3.3. Running another program

The teacher can use another program to test a different feature of the student’s program. Among other possibilities, this allows running static/dynamic analysis of the student code, e.g., the teacher can run checkstyle [1] to check the style of the student’s java code. The “Program to run” replaces the student’s program for another one in the test case that uses it.

Format: "Program to run = path"

Example:

_images/advanced_program_to_run1.png

Example of using “Program to run” statement

Note

If you plan to use a custom script as in the example remember to mark it in the Files to keep when running.

6.3.4. Program arguments

This statement allows sending information as command-line arguments to the student program or the “program to run” if set. Notice that this statement is compatible with the input statement.

Format: "Program arguments = arg1 arg2 …"

Example 1:

This example shows how to use the “Program to run” and “Program arguments” statements to check if the student’s program creates a file with a name passed as a command-line argument.

_images/advanced_program_arguments1.png

Example of using “Program arguments” statement and “Program arguments” statements

_images/check_file_exist_code.png

Code of the check_file_exist.sh script

Example 2:

The following example shows how the teacher can use the “Program to run” and “Program arguments” statements to evaluate a SQL query exercise using different data sets.

_images/advanced_program_arguments2.png

Example of using “Program to run” and “Program arguments” statements

Note

If you plan to use a custom script as in the example remember to mark it in the Files to keep when running.

6.3.5. Expected exit code

This statement sets the expected exit code of the program case execution. The test case succeeds if the exit code matches. Notice that the test case also succeeds if an output matches.

Format: "Expected exit code = number"

The next example that shows the possibilities of “Program to run” and “Program arguments” statements to execute different programs. The first case changes the name of a file, the second compiles the file, and the third run the resulting program.

_images/advanced_exit_code1.png

Example of using “Program to run”, “Program arguments”, “Expected exit code” statements

6.3.6. Variation

This statement set that the test case must only apply if the indicated variation was assigned to the current student. If the variation does not match then the test case is fully ignored.

Format: "Variation = identification"
_images/advanced_variation1.png

Example of using the “Variation” statement

[1]http://checkstyle.sourceforge.net/

For more details about VPL, visit the VPL home page or the VPL plugin page at Moodle.