MHCI PSLC Data Shop Project

1
MHCI PSLC Data Shop Project

• Final Design Presentation

2
The Team
Jason Hum
Sandi Lowe
Sam Zaiss
Meghan Myers
Jeff Wong
3
Outline

1. Background PSLC The Data Shop
2. The MHCI Project
3. Use Case Scenario
4. Prototype Demo
5. Implementation Timeline
6. Wrap Up

4
Outline

1. Background PSLC The Data Shop
2. The MHCI Project
3. Use Case Scenario
4. Prototype Demo
5. Implementation Timeline
6. Wrap Up

5
What is the PSLC?
PSLC
LearnLab
Data Shop
Collect
Process
Access
Pre- Defined
Free- Form
Export
6
PSLC Goals

• Further current education research
• Enable new education research
• Support collaboration
• Support 7 LearnLab courses

7
What is the Data Shop?
PSLC
LearnLab
Data Shop
Collect
Process
Access
Pre- Defined
Free- Form
Export
8
Outline

1. Background PSLC The Data Shop
2. The MHCI Project
3. Use Case Scenario
4. Prototype Demo
5. Implementation Timeline
6. Wrap Up

9
The MHCI Data Shop Project
Access
Pre-Defined
Free-Form
Learning Curves
Error Report
Problem Profile
Data Export
Session Browser
Timeline Viz
Behavior Graph
Help FX
Export
10
Project Requirements

• High-Fidelity Proof-of-Concept Prototype
• In-Depth Research Including Weekly User Testing
• 13 Contextual Inquiries
• 8 Requirements Interviews
• 12 Competitive Analyses
• 37 User Tests
• Deliverables
• Current Prototype
• Requirements Document
• Design Specification
• Supporting Data
• Design Iterations

11
MHCI Project Timeline
Summer Workshop
Start of Summer
End
UserTesting
Iteration
Hi-Fi Prototype
Paper Prototypes
Mid-Fi Prototypes

• Began With Low Fidelity Paper Prototypes
• Gradually Added Features Increased Fidelity
• Weekly User Testing Throughout

12
Project Themes

• Context Matters
• Facilitate Inter-Report Navigation
• Create Specialized Reports
• Emphasize Visual Communication

13
Outline

1. Background PSLC The Data Shop
2. The MHCI Project
3. Use Case Scenario
4. Prototype Demo
5. Implementation Timeline
6. Wrap Up

14
ITS Background

• Intelligent Tutoring Systems (ITSs) used
• To help students learn
• To gain insight into how students learn
• Consist of a series of problems in a particular
  subject.
• Order of problems is random
• Each problem composed of a number of steps each
  of which test one or more knowledge components

15
Meet Dave Jargenson
Name David Jargenson, Ph.D. Age 35 Affiliation
Research Scientist, Wisconsin Center
for Education Research

• Interested in how students learn Algebra.
• Has been doing education research for 10 years.
• The PSLCs newest member, he has already run a
  study with the Center.
• Trying out the Data Shop with a couple basic
  studies.

16
Daves Data Exploration
Determine Most Frequent Unexpected Error
17
Outline

1. Background PSLC The Data Shop
2. The MHCI Project
3. Use Case Scenario
4. Prototype Demo
5. Implementation Timeline
6. Wrap Up

18
Error Report

• Supporting Data
• Requirements Solicitation (VanLehn, Koedinger,
  Ritter)
• General Research Contextual Inquiries (U3, U8,
  U10)
• LearnLab Research Contextual Inquiries (U11, U12)
• Goal
• View all mistakes that students made, by
  frequency, steps and knowledge components

19
Error Report

• "Here we are. Errors by classification. Hmm,
  unanticipated? Oh, I can click it. Okay, the most
  common one is miles.
• - U30

20
Learning Curves

• Supporting Data
• Requirements Solicitation (Koedinger, Aleven,
  Ritter)
• General Research Contextual Inquiries (U2)
• LearnLab Research Contextual Inquiries (U13)
• Goal understand students performance over time
  particular knowledge components

21
Learning Curves/Problem Profiles

• Oh! I dont need to see the transaction table,
  its right here in the graph. (-U28)
• I like being able to see the curves without
  punching in the formulas. (-U1)
• I love that the as and the bs come right up.
  (-U36)
• I liked the Problem Profile. Leave it as it
  is.(-U28)

22
Sample Selector

• Supporting Data
• Contextual Inquiry Research U3, U15
• Requirements Solicitation Aleven, Koedinger
• User Testing Pre-test Questions U15, U16,
  U19-U24, U26, U28-U34, U36
• Goal Define multiple groups of students and
  compare their performance throughout the standard
  reports within the Data Shop

23
Sample Selector

• "so the only option I have is 'all students.'
  Ah, but I can edit this list. (U27)
• Oh, maybe I need to create a new sample. (U31)
• making the samples was fairly easy. (U29)

24
Data Export

• Daves Goal is to
• Select a subset of his data
• Export it to a file for further analysis
• Supporting Data
• Requirements Solicitation (Koedinger, VanLehn)
• General Research Contextual Inquiries (U5, U8)

25
Data Export Questions
26
Outline

1. Background PSLC The Data Shop
2. The MHCI Project
3. Use Case Scenario
4. Prototype Demo
5. Implementation Timeline
6. Wrap Up

27
Implementation plan
Estimated Implementation Time (in Days)
28
Outline

1. Background PSLC The Data Shop
2. The MHCI Project
3. Use Case Scenario
4. Prototype Demo
5. Implementation Timeline
6. Wrap Up

29
Context Matters

• Intimate Knowledge of Tutors Required
• Dig a little deeper, right away

30
Inter-report Navigation

• Reports are useful when they are connected.
• Carry context between reports when possible.

31
Specialized Reports

• More tailored than a stat package
• Eliminates grunt work

32
Use Visual Communication

• Get familiar with data quickly
• Identify points of interest

33
Evolution of Error Report
34
Evolution of Error Report
"I really don't understand what it error report
means. When Ken was showing it to us earlier in
the day at the Data Shop demo it made sense but
on my own I wasn't sure." User 13, Summer School
35
Evolution of Error Report
"This wasn't very helpful... probably just the
layout - it's hard to decipher.this is very
difficult to read Im not sure what these errors
mean." User 13
36
Evolution of Error Report
37
Evolution of Error Report
38
Evolution of Error Report
39
Evolution of Error Report
40
Acknowledgements
Bob Kraut
Ken Koedinger
Andrea Knight
Kurt Van Lehn
Vincent Aleven
Shipra Kayan
Carolyn Rose
Polo Chau
Michael Bett
Alida Skogsholm
Peter Centgraf
Braden Kowitz
Bonnie John
Ben Billings
41
The End
Its like having my very own grad student!
User 21
42
Backup Slide Data Visualization

• Information Visualization (Card, 2003) says that
  Data Visualization improves cognition in 6 ways
• Increasing the memory and processing resources
  available to users
• Reducing the search for information
• Using visual representations to enhance the
  detection of patterns
• Enabling perceptual inference operations
• Using perceptual attention mechanisms for
  monitoring
• Encoding information in a manipulable medium

43
Backup Slide - Error Report

• Horizontal Stacked Bars
• Option to take hints as errors of omission
• Allows them to compare down the line
• Error names fit better horizontally
• Visualization provides better performance than
  tables

44
Problem Profile

• Supporting Data
• General Research Contextual Inquiries (U1, U3,
  U7, U8, U9, U10)
• LearnLab Research Contextual Inquiries (U11, U12)
• Course Committee Survey (Chem)
• Think Aloud Pilot (U2)
• Goal Understand students performance on a
  particular problem, and the problems context

45
Multi-Selection(in long lists)

• A more standard method of indicating
  multiple-selection
• Highlight helps users quickly spot which items
  are selected if scrolling the list.
• if there were check boxes on the side I would
  have known I could select more than one (U15)

46
Scrubbing

• A method to quickly compare across knowledge
  components
• Oh interesting. It the next curve pops up.
  
• (U35)
• oohthat's so cute.I'm going to click on that
  point to see why it jumped back up.
• (U36)

47
Why Not Just Add Condition?

• Within-subject experiments require the capability
  for students to be assigned to multiple
  conditions
• Sample Selector allows for multiple groupings
  based on individual researchers units of
  analysis

48
What About Behavior Characteristics?

• The Sample Selector can build groups of students
  based on any characteristic in the database
• Student behavior characteristics are not
  currently fields in the database
• Once explicitly defined and included in the
  database, any behavior characteristic can be
  added and then used to build samples

49
Student Characteristics
50
Problem Characteristics
51
Step Characteristics
52
Data Export Needs Served

• Escape hatch
• Users can do whatever they want
• Narrow down data
• Export only rows which are relevant
• Export only columns which are relevant.
• User tables tend to be very wide.
• Users tend to copy out only the columns they need
  and move them to different worksheets.

53
Why choose columns?

• Columns are defined by Data Shop architecture
• All may not be relevant to every study
• You fit only the columns you are interested in
  within the width of the table.
• No side scrolling or rearranging of columns

54
Direct Manipulation

• Some direct manipulation of the table in Data
  Export
• Users 5, 11, and 27 tried to manipulate the graph
  directly.
• Having the pop-up menu instantly tells the user
  what clicking on the graph means.

55
Simple Filtering

• We allow simple filtering on each column
• Easy-to-use complex querying is an open problem
• Complex querying is better done with existing
  query languages such as SQL.

56
Main Screen
57
Filter Dialog
58
Export Dialog