PREVA

1
PREVA

• University of South Florida
• PRogram EValuation Assistant

or The Teaching Assistants Friend
2
PREVA

• The users of PREVA will be instructors and
  teaching assistants in programming courses
• At the moment, it is particularly suited for
  beginning courses up to Data Structures
• Testing its usability in other course is part of
  an ongoing effort
• The hoped-for target courses for this fall are C
  for Engineers, Program Design and Data Structures

3
Program Evaluation

• After downloading student programs, a teaching
  assistant or instructor has to do the following
  for each program
• Compile the program
• If successful, execute the program, possibly with
  several different inputs
• If no run-time errors occur, check the output to
  see if input/output specifications are met
• Check the source code for programming style and
  structure

4
Program Evaluation

• While all the phases of program evaluation can be
  time-consuming, the first two are particularly
  tedious
• Over the years, instructors (including myself)
  have constructed or refined scripts to automate
  the compilation, execution and, in some cases,
  output checking phases of evaluation
• PREVA attempts to provide such a system with a
  user-friendly, graphical user interface

5
PREVA

• The PREVA system runs on Unix systems via the
  X-Windows interface
• A later version will run on MS Windows
• It requires that Python and its attendant
  libraries have already been installed and are
  publicly available
• After initializing the system for a specific
  course, programming projects must be created in a
  particular format
• Both steps are straightforward

6
Overview

• System Goals
• Automate the tedious parts of programming
  project grading
• Make it possible to quickly identify programs
  with basic problems , like compilation and
  runtime errors
• Increase the amount of time available for graders
  to directly evaluate aspects that are not easily
  automated, like programming style and structure

7
Graphical User Interface

• The system provides a graphical interface for
• Course creation
• Project creation
• Distribution, compilation and execution of
  student programs
• Viewing results for all students in a database
• The initial window for my Spring 2008 Data
  Structures class is shown on the next slide

8
Initial Interface
9
System Use Course Creation

• Currently available on the CS Unix Servers for
  use by CSE faculty and graduate assistants
• The present system uses Blackboard for program
  submission via the Assignment feature
• Other options will be provided later
• Steps in Course creation
• Create and change to the course directory
• Start the system with the command/home/csee2/rti
  ndell/Python/preva

10
System Use Course Creation

• The system will detect the lack of a course at
  this time and present a course creation page

11
System Use Course Creation

• You will need to fill in the entries for the
  Course Name and the Semester
• Optional at this time enter the name of the
  tab-delimited gradebook from Blackboard in the
  Student Roll File
• After making your entries, click the button
  labeled Save Course
• If you add the roll file later, fill in that slot
  and click Save Course afterward.

12
System Use Project Creation

• The system maintains a separate archive of
  programming projects
• The clickable commands related to this archive
  are shown on the Archived Project page of the
  initial screen

13
System Use Project Creation
14
System Use Project Creation

• The components of a project are
• Supplied Files A list of files that you will
  supply Examples files needed for compilation
  (stack.h)
• Compile Commands A list of compile commands to
  be used, which might include separate
  compilation commands
• Execution Commands This allows for several
  executions with different input, etc.
• Output Comparision Commands Used if you do
  direct output checking

15
System Use Project Creation

• In the current version of PREVA, any files needed
  for the program, like supplied or correct output
  files, must be manually moved into the course
  directory
• To create a project, click Add New Project
• You will then see the following window
• Fill in the needed parts and hit OK

16
System Use Project Creation
17
System Use Project Creation
18
System Use Project Creation
19
System Use Project Creation
20
System Use Project Creation
21
System Use Project Creation
22
System Use Project Use

• When you add entries to the Supplied Files list,
  you can use absolute path names (/)
• If you just give a file name, the system will
  look for the file in the course directory

23
System Use Project Use

• Course projects are numbered
• Each such project is chosen from the list of
  archived projects
• This is done using the Add Project from Archive
  button

24
System Use Project Use

• If you have n projects at the time you add a new
  project, it becomes project number n1
• When you add the project, a program directory and
  individual student subdirectories are created
  automatically
• You must then move any needed files into the prog
  directory
• These might include supplied files, input files,
  execution scripts or correct output files needed
  for comparison

25
System Use Project Use

• You also have the choice of viewing the details
  of a project (on the Course page and on the
  Archived Projects page)
• On the Course page you have the option of
  replacing a numbered project with a new one
• There is no option for deleting a course project,
  since they are numbered and correspond to
  subdirectories

26
System Use Project Evaluation

• After initial Course and Project setup
• Download the zip file of submissions from
  Blackboard and place them in the program
  subdirectory
• Go to the Process Submissions page of the GUI
• Carry out the following steps in order
• Click on the Distribute Submissions button to
  have each students submissions placed in the
  students directory
• Click on the Process Submissions button

27
User Interface for Evaluation
28
System Use Student Database

• The student database is used to record very brief
  indications of the result of compilation,
  executions and possibly output checking of a
  program
• The advantage of the project results is that the
  grader has a clear indication of what to check
• Compiler Errors
• Runtime Errors
• Output Errors (when using output comparison)
• When no obvious errors occur, the grader can
  concentrate of program structure and style

29
System Use Student Database

• For each project, the result is shown as a string
  of letters, one for each execution of the program
  (provided compilation successful)
• The possible letters are
• c compiler errors
• r runtime errors
• x executed without runtime errors (used when
  no output comparison done)
• o output error (failed comparison)
• g Good - output comparison succeeded

30
System Use Student Database

• If you click on the Student Information tab, you
  will see something like this

31
System Use Student Database

• So, click on a students name

32
System Use Student Database

• Then click on View/Update a Students Results

33
System Use Student Database

• You can update any of the results directly in the
  form and hit Save Changes to send the result to
  the database.

34
System Use Student Database

• Other options View Project Results

35
System Use Student Database

• Other options View All Results

36
System Use Student Database

• Adding a new student to the database

37
Output Comparison

• There are some projects where automatic output
  comparison is possible
• This is probably an infrequent situation
• One example is where you provide functions to do
  all the output, so you can control the
  formatting, etc
• Another example is where the output files are
  very special
• Example data compression programs with very
  specific algorithms
• Binary file comparison is effective here

38
Output Checking

• If you cant do automatic output file
  comparisons, you want the output files to be
  easily checked by hand
• Providing Expected Output files for hand
  checking is effective in this situation
• The human checker doesnt expect a
  character-by-character match and can judge the
  correctness accordingly
• For these situations, you need to save the output
  files for visual inspection
• Another approach is to use try scripts

39
Addendum Try Scripts

• There are two ways to execute a compiled program
  several times with different inputs in PREVA
• The method built into PREVA is by using multiple
  Execution Commands (see the Project structure)
• In the Execution Commands section of the Project
  specification, give one line per execution.
• Example
• ./a.out lt ../../input1 gt output1
• ./a.out lt ../../input2 gt output2
• . . .

40
Addendum Try Scripts

• Alternate Method "try" scripts
• Dr Rollins Turner has used shell scripts he
  typically named try.
• In these scripts, the execution command is given
  with redirected input from within the script
  itself
• This is followed by echoing an expected output to
  the screen
• The structure is shown on the following slide

41
Addendum Try Scripts

• a.out lt -EndOfInputinput1 line1input1
  line2...input1 last lineEndOfInpuTecho
  echo
  Expectedecho line1 of outputecho line2 of
  output...echo last line of ouputecho
  a.out lt
  -EndOfInput

42
Addendum Try Scripts

• If the script on the previous slide, named try,
  is made executable and stored in the project
  directory, then the Execution Command section of
  the Project specification would have just one
  line
• ../../try gt output
• One disadvantage is that the inputs are not
  echoed in the output

43
Python Try Scripts

• We provide a utility to produce python scripts to
  do the same thing as the shell scripts
• These scripts have the advantage of showing all
  user input in the output file
• The utility prompts for the exectable command,
  then for input sections followed by expected
  output sections
• You may enter as many input-output sections as
  you like
• If the script is named try.py, the execution
  command would be ../../try.py gt output

44
Under the Hood

• Each course is kept in its own directory
• This directory contains
• A Course object
• A database of Project objects
• A student database containing evaluation results
• A directory for each project called prog1, prog2,
  

45
Under the Hood

• Each project subdirectory contains
• Files needed by the project
• A student subdirectory which contains
  subdirectories for individual students
• All compilation and execution takes place in the
  individual student directories
• As mentioned above, you will need to manually
  move any needed supplementary files into the
  course directory

46
System Use Help!

• I will be available to help set up and utilize
  the system
• Reminder - steps to start using the system for
  a course
• create the directory for the course on the CS
  unix system
• if available, download the tab-delimited
  gradebook from Blackboard
• enter the command /home/csee2/rtindell/Python/pre
  va
• Fill in the information and hit Save Course