Title: A Virtual TA for Computer Science 1
1 A Virtual TA for Computer Science 1 Cecily
Heiner Dr. Joe Zachary Advisor Arthur Collard
Advisee School of Computing
Approach
Abstract
Research Questions
In a recent talk on campus, Turing Award winner
Alan Kay identified mentoring as a very difficult
but very desirable goal for educational
technology. One step towards that goal is
extending the traditional intelligent tutoring
system model which assumes a single student on a
single, isolated machine to include the previous
elements plus an additional, more intelligent
user (e.g. a teacher) on an additional machine.
This extension to the model raises a number of
interesting research questions Should the
teacher or the software respond to the student?
If exchanges take place in an electronic
environment, can software "learn" by
"observation"? Can a piece of software be
written that has meta-cognitive awareness? What
other research questions can be answered with the
data gleaned from the interactions within this
electronic environment? We aim to answer these
questions with real data from an implemented
system that can be deployed in the introductory
computer science classes at the University of
Utah. The resulting system should lead to a more
scientific understanding of the problems and
difficulties that students face in introductory
courses. However, the greater gains will come in
understanding general principles of modeling
"educational workflows". 2
Use computers to improve education and do
research that cant be done without
technology. Build a system with interchangeable
parts that generalize to many domains
What kinds of questions do students ask? Which
questions can be delegated to a Virtual Teaching
Assistant? When should a Virtual Teaching
Assistant intervene? How should a Virtual
Teaching Assistant intervene How can we train a
Virtual Teaching Assistant to learn from
experience?
OARS Online Assistance Repository System
Goal Accurately diagnose student
problems Technical Challenge Automatically
process lots of data in various forms from
various sources including natural
language Approach Feature extraction and machine
learning exploit temporal and spatial
locality Example
Data Collection
Who Students in CS2010, CS2020, CS 1000, and
CS2000, plus their TAs and professors What
electronic communication over e-mail and a
proprietary system with chat and attachments plus
data logging software and priority queues for
question answering locally or remotely When Fall
2006-Fall 2007 Where University of Utah
NL Input NL Input Compiler Input Compiler Input Compiler Input Code Input Code Input
smoke the static void new
1 1 1 1 0
0 1 1 24 1
1 0 1 1 24 0
IRIS Instant Remedial Intervention System
Introduction
Goal Automatically deliver effective
interventions that increase student learning in
Introductory Computer Science classes Technical
Challenge Measure the learning gains from
individual interventions Approach Embedded
experiments comparing several interventions Exampl
e
Evaluation
Traditional Intelligent Tutoring Model
DB
Turing Test Can students distinguish between
human and computer delivered help? Student
Satisfaction Are students satisfied with help
from system? Intervention Evaluation Which
interventions increase student learning
most? System Evaluation Do students who use the
system learn more than students who dont?
ITS
Student
Researcher
Teacher
PS
- Problems with the Traditional Model
- Teachers and ITS compete for student attention
- Difficult to study student-teacher interaction
- Unnatural serial progression for students
- Students dont articulate well
- Difficult for teachers to create new problems
- ITS editors must be domain and AI experts1
Classification Information Information Evaluation Evaluation
subject source used TT SS
recursion Sun 5 2 2
new Java 1 3 1
animation TA 12 5 1
References
Proposed Intelligent Tutoring Model
- Koedinger, K.R., et al. Opening the Door to
Non-programmers Authoring Intelligent Tutor
Behavior. in Intelligent Tutoring Systems. 2004.
Maceio, Brazil Springer Verlag. - http//www.collide.info/ITS2006/
EYES Extending Your Evaluation Scaffolding
Student
ITS
Goal Provide accessible, low-risk
evaluation Technical Challenge Provide an
effective evaluation that doesnt give the answer
away or encourage laziness or procrastination
provide evaluations appropriate for the best and
worst students Approach Provide answer only when
student is within a small specified window of
ability Example Errors to Catch
Researcher
Teacher
- Related Research Goals
- Create an ITS that does automated QA and
- learns by observation
- has meta-cognitive awareness
- Improve student access to educational materials
- especially recently added information
- better automated problem diagnosis
- Spelling Errors in interface
- Methods that return wrong values