A Web-based Automatic Evaluation System

1
A Web-based Automatic Evaluation System

• Chittaranjan Mandal
• (School of IT, IIT Kharagpur, India)
• Chris Reade
• (Kingston Business School, Kingston University,
  UK)
• Vijay Luxmi Sinha
• (School of IT, IIT Kharagpur, India)

2
Automated Evaluation

• Motivation
• Other systems and choices
• Our design and choices
• Details of assignment set-up
• Conclusions and The bigger picture

3
Programming assignments (at IIT Kharagpur)

• 9600 to mark per semester
• Setting assignment (5hrs?)
• Distribution and collection (Automated by WBCM)
• Marking (_at_20m each) 3200hrs 20 people 20
  working days each

For sophisticated testing maybe 5 times this.
4
Automated marking is a must

• Additional Benefits
• automated rapid feedback (resubmission
  possibilities)
• consistency
• plagiarism checking
• rigorous testing (suite)

5
Different to simple test and assessment

• What can be automatically assessed?
• Program behavior (execution)
• Correctness testing
• Performance testing
• Whole program and/or components
• Data (randomising special test cases)
• Large numbers of runs
• Analysis of structure and style (still very
  hard to do)

6
Other Systems

• Full Automation versus Assistance
• examples Try (1989) / ASSYST 1997
• Feedback Only versus Full Management
• examples Submit (2003) / Ceilidh (1993)
• Eval of Design / Behavior / Components
• Fixed / multiple programming language
• Pascal, Ada, Scheme, C, C, Java
• Web-based?
• examples GAME (2004), Submit (2003)

7
Our design choices and goals

• Choose
• Component testing (performance and behavior)
• Single language (for now)
• Full automation AND rapid feedback
• Integration with full management system WBCM
• Security of the process
• Potentially unsafe programs (malicious/accident)
• Marking issues
• Partly working programs
• Feedback based on schema
• Address Overheads for assignment setup

8
Setting up an assignment

• Design a programming problem
• Sub-problem approach (white box)
• Marking scheme
• Implementation involves (formally)
• Expressing assignment plan (components and
  strategy)
• writing components for test harness
• describe testing data and process
• describe marking scheme with feedback
• example mergesort .

9
Mergesort
Student submission
Reference Files
main
main
make1
mergesort
mergesort
make2
merge
merge
Binary files to test
10
XML files in preference to web entry
Tester Script and files
XML specification source files marking
scheme testing process
processed
11
Input generation (types)
random integers (array / single)
(distinct / non-distinct) (un-sorted / sorted
ascending / sorted descending) (positive /
negative / mixed / interval) random floats
(array / single) (distinct / epsilon-apart)
(un-sorted / sorted ascending / sorted
descending) (positive / negative / mixed /
interval) strings (array / single)
(distinct / non-distinct) (fixed length /
variable length)
12
XML specification for input generation
lt!--------- input generation and test invocation
------------gt ltloop loopVar"x" start"1"
end"50"gt ltinput inputVar"a1"
type"integer varType"array"
sequence"ascend range"positive"
duplication"distinct"gt ltarray_sizegt50lt/array_
sizegt lt/inputgt ltinput inputVar"a2"
type"integer varType"array"
sequence"ascend range"positive"
duplication"distinct"gt ltarray_sizegt50lt/array_s
izegt lt/inputgt ltdistinction_classgt a1 a2
lt/distinction_classgt lttest_functiongt
test_merge_to_k (a1, 50, a2, 50)
lt/test_functiongt lt/loopgt lt!---------------------
--------------------------------------gt
13
XML specification for a test
ltauto_evalgt ltsource_filesgt . . .
lt/source_filesgt lttest marks"10"
abort_on_fail"true"gt lttextgt Evaluation of
merge_function lt/textgt . . .
lttestinggt lt!--- input generation and
test invocation ------gt
lt/testinggt lt/testgt lttest marks"10"
abort_on_fail"true"gt lt!--- similar
specification for testing mergesort ----gt
lt/testgt lt/auto_evalgt
14
Testing script flowchart
Submissions
Test Cases
Source File Details
Makefile
XML
Marking Scheme
Feedback Details
Reference Implementation
15
Conclusions

• This system is designed to support complex
  assessments. For automated assessment, the
  assessment set-up becomes the hardest part for
  designers
• Our tool directly supports implementation of
• Assessment strategy, marking sceme, feedback and
  testing harness
• Key features
• XML specs to generate testing script
• Re-use of assignment designs
• Future Work
• Generalising
• Assessing effectiveness with more examples