Marcos Almeida

1
Carnegie Learning Assistments Interface
Marcos Almeida Sue Young Chung Matthew
Easterday Andrea Knight Scott Robertson
2
Agenda

1. Project overview
2. Translating from Standards to Skills
3. Understanding skills
4. Remedy
5. Data driven design
6. Demo
7. Evaluation and Next Steps

3
Assistments Project

• Web-based computer tutor Assistment system
• that will help students prepare
• for standardized mathematics tests.
• Why important
• No Child Left Behind Act (accountability,
  consequences)
• Effective use of classroom time
• Goals
• predict a students score on a standardized test
• provide feedback to teachers about how they can
  adapt lessons to address gaps in students'
  knowledge

4
Our Project

• Design teacher interface so they see where
  their students need help.

5
The Challenge
A Standard 8.N.10 Estimate and compute with
fractions (including simplification of
fractions), integers, decimals, and percents
(including those greater than 100 and less than
1).
6
The Challenge
A Standard 8.N.10 Estimate and compute with
fractions (including simplification of
fractions), integers, decimals, and percents
(including those greater than 100 and less than
1). A Skill Translation (Symbolic to
Visualization)
7
A Skill

• Skills are the steps taken to solve a problem.

8
The Problem Space
9
Reporting Category
State Standard
Questions
Skills
10
Project Goals

• Communicate student and class progress
• Provide guidance for remedy

11
Our Solution

• Make the relationship between standards, skills,
  and questions explicit.
• Provide different views of the data to support
  teachers various uses.

12
Three Focus Questions

• How do teachers understand skills? What can we do
  to help them understand skills?
• 2. How to translate from the standards (that
  teachers understand) to skills (operational
  knowledge that the students have and that CL has)
  
• 3. How do teachers diagnose/evaluate and then
  what do they to remedy?

13
User Studies
We talked to 18 algebra/geometry teachers who use
CLs Cognitive Tutor and/or prepare their
students for the PSSA
14
Understanding Skills

• Interviews and Card Sort
• What we learned
• Teachers can identify what skills mean by labels
• Consensus on grouping of skills (what the
  correlation was)
• Non-consensus in labeling of group
• Teachers naturally break down questions into steps

15
Understanding Skills

• Important for us to make mapping from standards
  to skills very explicit
• Skill to question relationship is crucial
• Skills should be illustrated as a step within a
  question
• Definitions of skills should include examples

16
Translating from Standards to Skills

• Interviews and Prototype Think Alouds
• What we learned
• Standards are more concrete
• Evaluated against standards
• Practical needs to see progress
• Teachers get a feel of how their students are
  doing
• Data loses meaning without the personal context

17
Translating from Standards to Skills

• Need to explain skills in terms of standards
• Important for us to provide context
• Help teachers link Assistments to other sources
  of information / remedy

18
Remedy

• Interviews, Think Alouds, Background
  Questionnaires
• What we learned
• Teachers jump to remedy from performance reports
• Inexperienced teachers (under 3-5 years
  experience) were unable to generate remedy
  methods
• Teachers value class interaction to gauge if
  remedy is needed
• Re-use questions for quizzes, exams, and practice
• Most commonly assign more practice problems

19
Remedy

• Dependant upon first two focus being a success
• Trust Assistments analysis
• Provide guidance for remedy
• Provide support for class or student interaction
• Allow questions to be accessed for quizzes,
  exams, and practice problems

20
Design Activity
Aggregate Prototypes
Skills
Standards Relationship
Remedy
21
First Second Iterations
22
Feedback on Prototype

• Assistments data is not currently something
  teachers would know how to use.
• "I would go back and teach the part again (U14,
  U22)
• "I would assign more worksheets" (U20)
• Practical Needs
• What should I address next (U14, U15)
• Teachers sense of standards is more concrete
  than skills
• Have I covered all the standards
• Where in the standards are they stuck (U15,
  U20)

23
Feedback on Prototype
Individual Questions give teachers concrete idea
on where the students struggle Like most missed
question displayed (U14, U20) Good visual
presentation important for good testing
results this has stuff all over the place
(U15) What does performance mean? Does higher
number mean good or bad? (U14, U15, U20)
24
(No Transcript)
25
Design Requirement Doc

• Evaluation of designs
• Formation of designs
• Support future design for client
• (support our designs)

26
Requirements Requirements Requirements   support
1. Class Element 1. Class Element 1. Class Element U6 Teacher does not concentrate on individual student needs but examines needs of class at a whole
Goals 1 Assign homework2 Determine what to teach next3 Answer will the kids pass the test (will I get fired?)
1.1 User should be able to see what their kids need to know and prioritize skills Allow teachers to differentiate between skills that should be mastered versus skills that only require exposure (U2)U9 allow teachers to highlight particular skills for more information to be gathered by system (give students more problems/practice) by weighting them heavier and having them show up more often in problem line up (diagnostic, not remedy) 2.1
1.2 Users should be able to draw comparison between classes U1 provide task analysis and measurements/metrics of improvement for comparing classes between curriculums to aid in lesson study 6.3U6 DI Cares about seeing the class average
27
(No Transcript)
28
(No Transcript)
29
Explain skill in context of a question
Immediately view problem areas
30
A New Design

• Most importantly
• How many of my students will pass?
• Different tabs show further details
• Performance on state standards
• Prioritized list of skills needing improvement
• Progress
• Questions the class missed the most
• Could be extended with
• Comparison between classes
• Skill-Standard relation
• Remedies

31
Where we are
32
Where we are
33
Demo

• Student progress
• Communicating skills and standards
• Predict a students score on a standardized test
• Provide feedback to teachers about gaps in
  students' knowledge

34
Future Direction
Remedy