5. Advanced features

This chapter is intended for teachers that want to take full control of the VPL features. Teachers here will see how to customize the running and assessing process, and more.

5.1. Execution files

Execution files are a set of files that participate in the run, debug and evaluate tasks. This includes scripting files, program test files, data files, etc. These files will go with the student’s submitted files to run in an execution server. In the case of a student’s file has the same name that an “execution file” the “execution file” is used. Some of the files have a predetermined purpose based on their name: vpl_run.sh, vpl_debug.sh, vpl_evaluate.sh, and vpl_evaluate.cases.

The scripts vpl_run.sh, vpl_debug.sh, and vpl_evalaute.sh if set, replaces the default corresponding action. These scripts carry out the compilation or preparation phase of the action. Their aim is to generate a file named vpl_execution or vpl_wexecuton. These files must be a binary executable or a script beginning with “#!/bin/sh “. The system launchs vpl_execution in a textual terminal and launchs vpl_wexecution in a graphic terminal. The non-generation of one of these files impedes running the selected action. The vpl_debug.sh can not generate a vpl_wexecuton file.

The file vpl_evaluate.sh defines the test cases used by the VPL input/output evaluation program. See Automated program assessment for more details.

Execution files setting has as interface an IDE that allows defining the files. This interface provides the predefined files named above and the run, debug and evaluate buttons allowing to tests the effect of the “Execution files” in the last submission of the teacher.

Execution files example

Note

Execution files may be affected by the The based on feature.

5.1.1. Files included and excluded

For security reason the run action removes “vpl_debug.sh”, “vpl_evaluate.sh” or “vpl_evaluate.cases” from the task. The debug action removes “vpl_evaluate.sh” or “vpl_evaluate.cases”.

All execution tasks includes two auxiliar scripts: “common_script.sh” and “vpl_environment.sh”.

The “common_script.sh” script defines auxiliary functions for the other scripts. The “vpl_environment.sh” define shell environment variables with information about the task. The run, debug or evaluate script can use the following variables:

• LANG: used language.
• LC_ALL: same value as LANG.
• VPL_MAXTIME: maximum execution time in seconds.
• VPL_FILEBASEURL: URL to access the files of the course.
• VPL_SUBFILE#: each name of the files submitted by the student. # Ranges from 0 to the number of submitted files.
• VPL_SUBFILES: list of all submitted files separated by space.
• VPL_VARIATION: the identification of the variation assigned or empty.
• VPL_VARIATION + id: where id is the variation order starting with 0 and the value is the identification of the variation assigned. These vars have a sense when using :ref: the based on feature.
• If the action requested is evaluation, then the following vars are added too.
• VPL_MAXTIME: max time of execution in seconds.
• VPL_MAXMEMORY: max memory usable.
• VPL_MAXFILESIZE: max file size in bytes that can be created.
• VPL_MAXPROCESSES: max number of processes that can be run simultaneously.
• VPL_FILEBASEURL: URL to the course files.
• VPL_GRADEMIN: Min grade for this activity.
• VPL_GRADEMAX: Max grade for this activity.

5.1.2. pre_vpl_run.sh

This script gives teachers the power to create files programmatically before starting a task. If present in the execution files it is run by the common.sh script. The target of this script is to prepare the run, debug, and evaluate actions without changing the default script. This script can be used to create and populate DBs, create data files, create or modify the vpl_evaluation.cases files, etc.

5.2. Execution resources limits

You can set limits for the execution time, the memory used, the execution files sizes and the number of processes to be executed simultaneously.

These limits are used when running the scripting files vpl_run.sh, vpl_debug.sh and vpl_evaluate.sh and the file vpl_execution or vpl_wexecution built by them.

Resource limits example

Note

Resources limits may be affected by the The based on feature.

5.2.1. How to measure the required resources for an execution in an execution(jail) server?

In general is correct to leave the defaults value of resource limits. If you want to restrict or extending the resources used in a particular activity, it is recommended testing a standard solution of the activity, varying the resource from higher to lower to find the correct value.

5.3. Files to keep when running

For security reasons, after running the scripting files vpl_run.sh, vpl_debug.sh or vpl_evaluate.sh, and before running the file vpl_execution or vpl_wexecution built by them, the unneeded “execution files” are removed. If some of these files need to be when running the student’s program, the teacher must mark these files here, g.e. auxiliary libs or data files.

Files to keep when running setting example

5.4. Variations

A set of variations can be defined for an activity. These variations are randomly assigned to the students.

Here you can indicate if this activity has variations, put a title for the set of variations, and to add the desired variations.

Variations setting example

Each variation has an identification code and a description. The students see the description of their assigned variation at the task description. The identification code of the student’s assigned variation is set in vpl_enviroment.sh file to participate in execution process.

Variations assigned example

Note

The variations are affected by the The based on feature.

5.5. Check execution servers

This report checks and shows the status of each execution server available for this activity. The report also shows the tasks running or just finished in the current course.

Check execution server setting example

5.6. Local execution servers

VPL allows the setting of several execution servers. For each task requested the system selects one of these execution servers to carry out the task. The local execution server allows adding new execution servers for the current activity. This will allow having more execution power and fail tolerance for specific activities. It is also possible to set the only servers that will participate in the current activity by ending the list of local servers with a line containing “end_of_jails”. This allows having specific execution servers for specific activities that may have particular software needs.

Local execution server setting example

Note

Local execution servers may be affected by the The based on feature.

5.7. The based on feature

This powerful feature allows activities to inherit the options and files of other VPL activities. This feature enables the development of generic activities that can be used as a basis for others. In many cases, the generic activities are not used as activities to be used by students, their purpose is to establish a common framework for other activities. The following items are inherited from the selected activity as based on:

• Description. The “based on” description is inherited. If the current activity contains a description it is appended to the “based on” activity.
• Resource limits. The “based on” resource limits are inherited. Any resource limit set in current activity replaces inherited value.
• Upload file size limits. Current activity can replace inherited value.
• run/debug script. Inherits the selection in “:Execution options of run script and debug script if not set in the current activity.
• Execution files. All “execution files” are inherited. Current activity “execution files” add files and replace the ones inherited with the same name. Exceptionally, predefined scripting files (vpl_run.sh, vpl_debug.sh, and vpl_evaluate.sh) are append to the inherited.
• Local servers. Inherits local servers and append local servers in current activity.
• Variations. Inherited and current variations are used to generate multi-variation activities.

5.8. Adding support for a new programming language

Adding support for a new programming language in VPL requires two things: the programming language compiler/interpreter ready to be used in an execution server, and the scripts that allow preparing the student’s files to run. The first requirement can be done following the instructions of the selected programming language tool. The second is done by customizing the vpl_run.sh script for running and the vpl_debug.sh for debuging. If planning using the Automated program assessment customize vpl_evaluate.sh is unneeded. For more details of how create/modify theses scripts see Execution files.

A simple way to start your own script is to take the default script of another similar programming language. The default scripts are located in “/mod/vpl/jail/default_scripts/” of the VPL plugin source code.

The purpose of the script is to compile or prepare the implementation of the action requested run/debug/evaluate. If the script succeeds it will generate an executable file called vpl_execution to be run on a textual terminal or vpl_wexecution for execution in a graphics terminal.

If you have a predefined activity customized to run a new programming language. All the new activities configured as “based on” the predefined one will run also the new programming language. See The based on feature for more details.

5.9. Customizing automatic program assessment

VPL is a flexible tool such that you can customize the automatic evaluation process. Following the Execution files indications you can runs your own tools to evaluate the student’s code.

The customized evaluation must generate an output that follows the format below.

5.9.1. Evaluation output format

When an automatic evaluation is done, the system process the execution output (standard output) to obtain the comments (feedback) and proposed grade.

The comment (feedback) format is as follows:

A comment (feedback) line would be a line starting with “Comment:=>>”. A block comment would be contained between a line containing only “<|--" and other with "--|>”. The proposed grade is taken from the last line that begins with “Grade:=>>”.

If the automatic assessment is set, then the proposed grade becomes the final going to the Moodle grade book.

5.9.2. Formatting the comments (feedback)

The comments in the activity assessment has its own format:

• Lines beginning with “-” are titles.
• Lines starting with “> ” are preformatted text “<pre>” .
• The rest of the lines are considered content related to the previous title.
• Expressions of the form filename:number generate a hyperlink to the corresponding line of the file.
• At the end of each title line, optionally, you can add a negative discount. This value represents the discount attributed to the associated comment. This discount is never displayed to the students. This discount is used for the “calculate” button. Example:

- Error: infinite loop (-10)

5.10. Details of running a task

This section will try to explain what is happening in backstage of a running task.

Connections when running task:

1. User clicks run button. Browser send request by === AJAX (json) === > to the VPL plugin in the Moodle server.
2. VPL plugin in Moodle Server prepare task and send data by === http/https (XMLRPC) === > to a selected execution server.
3. Execution Server starts the task and returns to VPL plugin in Moodle server the task identification.
4. VPL plugin in Moodle server returns to the browser the task identification.
5. The browser monitorize the task by connecting by === ws/wss === > to the execution Server.
6. The browser may connect with the running program by connecting by === ws/wss === > to the execution Server.

Process of a student’s program execution (summarized). The following steps are performed:

1. The system takes the files submitted by the student
2. The system takes the files set by the teacher in “Execution files”. These files replace files of the student with the same name.
3. Depending on the action (run, debug, or evaluate) the system takes the customized or default script by detecting the programming language used based on the extension of file names.
4. In the case of evaluation, if there is no custom script, the VPL program for automatic assessment is also added. This program is based on the input and output of the program and requires that you specify the test cases in the file “vpl_evaluate.cases”.
5. These collected files are sent to an execution server.
6. The plugin informs the browser that the execution has begun.
7. If the request is an evaluation, when the task is finished, the result of the evaluation is retrieved from the execution server.

The VPL execution service is responsible for receiving and controlling the execution of code. The execution can be terminated for four reasons:

• Finish their execution normally.
• They are stopped when they deplete their assigned resources (time, memory, etc.).
• They are stopped on user request (e.g. the user closes the browser). This monitoring is done through a WebSocket connection from the browser to the jail server.
• The Moodle server requests the stop of the task. Each student can only have one running task, if the user requests to execute another task, the previous one will be stopped.

After the end of a task, the work area used is cleaned.

For more details about VPL, visit the VPL home page or the VPL plugin page at Moodle.