Labrador: A Tool for Automated Grading Support in Multi-Section Courses

1
Labrador A Tool for Automated Grading Support in
Multi-Section Courses

• Drexel University Programming
• Learning Experience (DUPLEX)
• http//duplex.mcs.drexel.edu
• Christopher D. Cera, Robert N. Lass,
• Bruce Char, Jeffrey L. Popyack,
• Nira Herrmann, Paul Zoski
• Drexel University
• Mathematics and Computer Science

2
Roadmap

• Introduction
• Problems and Solution Goals
• Labrador
• Discussion

3
The Group

• Bruce Char
• Professor, Computer Science
• Nira Herrmann
• Professor and Head, Mathematics
• Jeffrey L. Popyack
• Associate Professor, Computer Science
• Paul Zoski
• Instructor, Math and Computer Science
• Christopher D. Cera
• Computer Science Graduate Student
• Robert N. Lass
• Computer Science Undergraduate Student

4
Who Am I

• First TA in MCS to experiment with WebCT in
  December 2000
• Courses I have TAed using WebCT
• 3 x Introductory Programming I, II
• 1 x Object Oriented Programming
• Migrated course content to WebCT from previous
  course website
• HTML assignments and labs into question database
• Gradebook maintainer
• Wrote demos and documentation to train other TAs
• Developer of software supplements to WebCT to
  support course administration

5
The Course

• Intro Programming I, II
• Variety of Majors
• Computer Science
• Computer Engineering
• Information Systems
• Mathematics
• Digital Media
• Class
• 1 x 1 hour lecture
• 2 x 1 hour labs
• People
• 250-300 Students
• 2-3 Professors
• 10-12 Teaching Assistants

6
The Duplex Project An Overview

• Take advantage of advances in Information
  Technology to improve instruction and reduce
  costs for computer programming courses
• Modular Structure
• Multiple Entry Points
• Multiple Audiences
• Multiple levels of knowledge (Blooms Taxonomy)
• Computer Supported Cooperative Work (CSCW) in
  student labs
• Online Services Todays Topic

7
Roadmap

• Introduction
• Problems and Solution Goals
• Labrador
• Discussion

8
Before WebCT

• Hundreds of people involved in paper exchanges of
  handwritten assignments and quizzes
• Testing programs required floppy exchanges
• No Chat
• No newsgroup-style threads
• Feedback and grades are not online

9
Before WebCT

• Large Classes
• Managing several hundred students is difficult
  due to the logistics involved with organizing
  paperwork and people.

10
WebCT

• Service from Information Resources Technology
  (IRT) department (v3.1)
• MCS started Dec. 2000 (v3.5)
• Upgraded to v3.7 recently
• Transition to v3.8 planned in Fall

11
Noteworthy Features

• General Course Website
• Labs Online Quizzes with Automated Grading
• Homework Online Assignments
• Joint Staff--Student Chat
• Discussion Threads

12
Large Classes

• Still some difficulties
• Bulk download of assignment files and quiizzes
  for grading
• Files submitted in various compressed, archived,
  and encoded formats, and are tedious to unpack
  manually.
• Handling different sections required lots of
  searching and clicking.

13
Software Design Goals

• Bulk Assignment Downloading (prior to 3.7)
• Bulk Quiz Downloading
• Section Sorting for bulk downloads, or only one
  section

14
Software Design Goals

• Post-Processing student files
• Archive Extraction (tar, zip)
• Decompress (gz, zip)
• Decode (uue)
• Minimal staff intervention when transferring
  submissions between systems, eg. Plagiarism
  Detection Systems (PDS)
• Automatically collate source code
• Generate electronic documents to facilitate
  grading and archiving
• Remote Execution
• downloading to computer x (on campus) while
  operating at computer y (off campus)

15
The Bigger Picture

• Course-Specific Tasks
• Not all processing should be done on the server
• Select files have to be transferred to a
  different system for further processing and
  analysis by staff
• Client-side support is needed to perform this,
  preferably automated and not necessarily using a
  web browser.

16
Roadmap

• Introduction
• Problems and Solution Goals
• Labrador
• Discussion

17
Labrador Our Solution

• Client-side WebCT Supplement
• Implemented in Perl
• Works for users with TA and Designer access to
  WebCT

18
Perl

• Practical Extraction and Reporting Language
• Created in 1987 to replicate Unix shell
  functionality
• Powerful text manipulation
• High-level rapid prototyping and development
• Large library of modules and active development
  community
• Perl is used by many applications, including WebCT

19
A Taste of Labrador Perl Programming
20
Labrador works on Windows, Mac, Linux, Other Unix
Variants, etc.

• Perl has been ported to all major operating
  systems
• Perl programs can be compiled
• No need for the end user to install Perl
• Perl2exe for Windows support (www.perl2exe.com)

21
User Interface

• Different users / Different UI preferences
• Command-line
• Interactive
• Configuration File
• GUI (in development)

22
Required Info

• http//webct.drexel.edu/SCRIPT/CS164_Fall2002/scri
  pts/designer/dropbox_edit.pl?DROPBOX_ASSN_VIEW_si
  de_nav1006285909
• Username/Password
• Server URL webct.drexel.edu
• Course ID CS164_Fall2001
• Submission Name or ID 1006285909
• Optional Username List

23
Components
JPlag
Submission Downloader
Section Sorting
Post Processing
PDF Generator
Moss
24
Bulk Downloader

• Web-Crawler simulates clicks of actual staff
  member
• Downloads actual web content (HTML)
• Parses HTML to find desired text or URLs to crawl
  to next
• Uses HTTP operations
• Works on assignments and quizzes
• Only component in Labrador which interacts with
  WebCT

25
Organizing Submissions by Section

• Not supported in original Labrador prototype,
  added later on.
• Organizes files into directories
• 3 possible input methods
• Creating a Section column in gradebook
• Username, Section CSV File
• Username file

26
PDF Generation for Electronic Mark-up

• Adobe Portable Document Format can be viewed on
  all major platforms
• With Adobe Acrobat, PDFs can be annotated by
  graders
• Sony VAIO Slimtop PC (PCV-LX920)
• Wacom Pen Tablet.

27
VAIO / Wacom
28
PDF Markup Example
29
Redistribution

• How can we return annotated PDFs back to
  students using WebCT?
• WebCT Mail
• WebCT Groups
• Version 3.8 now addresses this issue

30
Heterogeneous Systems

• Different systems want different formats
• Plagiarism Detection Systems
• Moss has been used most extensively
• Began experimenting with JPlag more recently
• Future work Automatic program compiling and
  testing

31
Automated Plagiarism Detection

• Digital formats make "borrowing" easy
• Browsing similar works needs a simple and quick
  user interface.
• Careful review by faculty to assess results and
  present to students

32
Moss

• Processes C, C, Java, ML, Lisp, Scheme, Pascal,
  and Ada programs.
• Common code feature reduces false positives

33
Moss Interface
34
JPlag

• Processes C, C, Scheme, and Java programs
• For plain text files, it matches a user specified
  number of words appearing in succession
• Could be used for any course grading written
  (text) documents

35
Command Line Invocation
36
Interactive
37
Quiz Download with Config File
38
Roadmap

• Introduction
• Problems and Solution Goals
• Labrador
• Discussion

39
Recent WebCT Enhancements

• Relevant to this talk
• 3.7
• Addressed bulk download issue for assignments
• 3.8
• Attaching documents to an assignment

40
Future WebCT Enhancements

• Power users will need functionality not yet
  supported
• Every domain will also require additional
  functionality
• Not feasible for all domain-specific
  functionality to run on the WebCT server

41
Wanted

• API for non-administrators
• Stateful protocol so clients can be built by 3rd
  parties

42
HTTP Insufficient Protocol

• Inconvenient content interchange
• Heavy client interaction
• A HTTP based approach will be sensitive to the
  exact location of web pages, and format of text
  within them.

43
Labrador Availability

• Contact Us
• http//duplex.mcs.drexel.edu

44
Project Support

• The Pew Learning and Technology Program at the
  Center for Academic Transformation
• National Science Foundation, Division of
  Undergraduate Education, DUE-0089009
• The Ramsey-McCluskey Family Foundation, Margaret
  Ramsey, '84
• Drexel University

45
References

• 1 Alex Aiken. Moss A system for detecting
  software plagiarism (unpublished),
  http//www.cs.berkeley.edu/aiken/moss.html.
• 2 Jeffrey L Popyack, Bruce Char, Paul Zoski,
  Nira Herrmann, and Christopher D. Cera. Managing
  course management systems. In Proceedings of the
  thirty-third SIGCSE technical symposium on
  Computer Science Education, Birds-of-a-Feather
  Sessions, page 423, 2002.
• 3 L. Prechelt, G. Malpohl, and M. Philippsen.
  Jplag Finding plagiarisms among a set of
  programs. Technical Report 2000-1, Fakultat fur
  Informatik, Universitat Karlsruhe, Germany, March
  2000.
• 4 Larry Wall, Tom Christiansen, and Jon Orwant.
  Programming with Perl. OReilly and Associates,
  3rd edition, 2000.
• 5 Michael J. Wise. Yap3 Improved detection of
  similarities in computer program and other texts.
  In Proceedings of the twenty-seventh SIGCSE
  technical symposium on Computer Science
  Education, pages 130134. ACM Press, 1996.

46
Roadmap

• Introduction
• Problems and Solution Goals
• Labrador
• Discussion
• Bonus Slides

47
Blooms Taxonomy

• Knowledge
• remembering of previously learned material
  recall (facts or whole theories) bringing to
  mind.
• Comprehension
• grasping the meaning of material interpreting
  (explaining or summarizing) predicting outcome
  and effects (estimating future trends).
• Application
• ability to use learned material in a new
  situation apply rules, laws, methods, theories.
   
• Analysis
• breaking down into parts understanding
  organization, clarifying, concluding.
• Synthesis
• ability to put parts together to form a new
  whole unique communication set of abstract
  relations. 
• Evaluation
• Ability to judge value for purpose base on
  criteria support judgment with reason. (No
  guessing).

48
Hurdles

• Automating course management and delivery to ease
  burden on faculty
• Submission and return of assignments
• Electronic grading of assignments with written
  comments
• Automated compiling / running of programs
• Online support for students
• Online quizzes and assessment surveys