Designing and Using an

1
Designing and Using an Embedded Assessment System
to Track Student Progress
Andrew Galpern, Graduate Student
Researcher Stephen Moore, BEAR Center Associate
Director Berkeley Evaluation and Assessment
Research Center UC Berkeley - Graduate School of
Education
2
Overview

• Welcome and introductions
• The BEAR Assessment System
• Four principles of assessment
• Teacher activities and discussions
• Closure and conclusions

3
Introductions around the room

• Your name?
• Where are you from?
• What subject and grade do you teach?
• One question you have about assessment?

4
Teacher group activity 1 Scoring
Read the first example of student work and
note to yourself what the responses tell you
about student understanding. We go to the
moon and do these other things, not because they
are easy but because they are hard.
John Kennedy
5
Embedded Assessment

• Assessment activities integrated within
  instructional activities
• Teachable moments ltgt Assessable moments
• And now for something completely different
  Monty Python

6
BEAR Assessment System

• Four principals
• Construct is defined developmentally
• Assessment is matched to instruction
• Student responses are linked to development
• Evidence is based on a measurement model

7
Connection to NRC Assessment Triangle
8
Developmental Construct
Match Assessment and Instruction
Evidence Based on Measurement Model
Outcome Space Responses Linked to Development
9
Principle 1 The construct to be measured is
defined developmentally Identify the levels of
development in your construct.

• Not just adding more knowledge
• Also adding connections, relations, strategies,
  and organization
• Progressing through qualitatively different
  levels of understanding

10
A Generic Developmental Construct
Students have a lot of what you are measuring,
a strong understanding, sophisticated
explanations, advanced, etc. Students have a
moderate amount of what you are measuring, a
moderate understanding, moderately complicated
explanations, proficient, etc. Students have a
small amount of what you are measuring, a
beginners understanding, basic explanations,
emerging, etc.
High Medium Low
11
Why things sink and float (FAST Buoyancy
Curriculum)
Relative Density Density Mass Volume Mass
or Volume
High understanding Medium understanding Low
understanding
12
Teacher group activity 2 Construct a
Developmental Construct

• Pick a science concept
• Create a developmental construct map, showing
  levels of development from low to high
• Developmental and Cumulative

13
A Developmental Construct Map Quantity of
understanding Qualitative levels of
development
Level X -- -- -- Level 1
High Medium Low
14
Principle 2 Assessment matches
instruction. Identify assessment opportunities
within ongoing instructional activities.

• Assessable moments
• Non-stop?
• What will you observe?
• How will it be recorded?

15
Student buoyancy activity FAST Curriculum
Predict, Observe, and Explain (POE)
Observe Explain
16
Teacher group activity 3 Match Assessment to
Instruction

• Think of a nice instructional activity to teach
  the construct you created.
• Create a list of ways to observe what students
  know or can do within the activity.
• How to embed these in the activity?
• Questions? Multiple choice, open-ended
• Other formats? Notebooks, drawings,
  demonstrations, interviews

17
Principle 3 Student assessment responses are
linked to development. Define the Outcome
SpaceIdentify how student responses map onto
your construct.

• Scoring guide or rubric
• What responses indicate what level of
  understanding?
• Must the levels in a scoring guide for a given
  assessment activity (item) be identical to the
  levels of your construct?

18
Teacher group activity 4 Scoring againthis time
with an example scoring guide

• Score student work based on the scoring guide
  provided.
• Compare and discuss your scores.
• Change/Edit scoring guide if needed.
• Change/Edit construct if needed.

19
Principle 4 Evidence is based on an appropriate
measurement model. Base assessment conclusions on
a valid and reliable scale of measurement with
known psychometric properties.

• IRT Item Response Theory
• Teacher-friendly software
• How do the students and the assessment activities
  line up on the scale of measurement?

20
WARNING! You are about to see the UGLIEST (and
yet the MOST USEFUL) slide in this entire
presentation. Viewer discretion is advised.
21
Students and items on the measurement scale
defined for the construct
WRIGHT MAP CAESL/FAST (MLE)
Dimension WTSF ASSESSMENT RL_at_4 Map of
MLE estimates and response model parameter
estimates

students Thurstonian
Thresholds --------------------------
--------------------------------------------------



wrl4b.RD
3
wrl4a.RD





wrl4a.D wrl4b.D
2

X

wrl4a.V wrl4a.MV
wrl4b.V
wrl4b.MV
XXXwrl4c.RD

Xwrl4d.D
1 Xwrl4d.MV

wrl4c.D
XXXXwrl4d.V

XXXXX
XX

XXXXXXXXXXwrl4d.M
0 XXXXXXXXXXXXXXXXXwr
l4c.V wrl4c.MV
---XXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXwrl4b.M wrl4c.M

XXXXXXXXwrl4a.M
XXXXXXXXXwrl4a.PM
wrl4d.PM -1
XXXXXXXXXXXwrl4a.UF wrl4c.PM

XXXwrl4b.PM
XXX

Xwrl4d.UF
wrl4c.UF

X
-2
wrl4b.UF
wrl4c.OT

wrl4b.OT


wrl4a.OT
-3
wrl4d.OT






Each X
represents 2 students, each row is 0.179 logits

Difficulty estimates for all the items
Ability estimates for all the students
22
A students location on the scale of measurement,
shown relative to the scoring levels for selected
items
23
Progress over time for one student
Embedded Assessments
24
Progress over time for all studentsPre-test
versus post-test results
25
Teacher group activity 5 Evaluate a conclusion
about students ability, reached using assessment
information
Q What color is your parachute? A Anything
but red.
26
The Parachute Packers a thought experiment
Monday Count and label all the parts. Quiz
1 Tuesday Explain the function of each part.
Quiz 2 Wednesday Memorize the steps. Quiz
3 Thursday Pack the materials. Quiz 4 Friday
Test your chute! Final exam
27
The Parachute Packers Assessment results
What grades would you give them? Who do you want
packing your parachute?
28
Percentomatic!
Whats wrong with a percent-correct score?
Performance when it counts.
29
Timing and Frequency of Assessments Choose a
balance that fits the instructional setting.
Formative vs. Summative Timing and
Frequency Depth Format Half-life
30
Teacher group activity 6 Discuss the
advantages/disadvantages of various scenarios for
frequency and timing of assessments.
pre-test
mid-term
final
31
pre-test
mid-term
final
1 2 3 4 5
32
Designing and Using an Embedded Assessment System
to Track Student Progress
BEAR Center UC BerkeleyGraduate School of
Education 2000 Center Street, Suite 301 Berkeley,
CA 94720-1680 Phone [phone] Fax
510 666 9286 Email bearcntr_at_berkeley.edu Grad
eMap software online demo http//bearcenter.berke
ley.edu/kennedy/GMOnline/