Increasing the Payoff: Getting the most out of CBEs

1
Increasing the Payoff Getting the most out of
CBEs

• Michelle Lewis
• mlewiswa_at_yahoo.com

2
Purpose

• Build highly qualified school teams skilled in
  the use of curriculum based evidence to assess
  reading comprehension and guide instruction
• Extend prior learning

3
Products

• Scoring protocol
• Protocol for selecting anchors
• Data collection and grading tools

4
Process

• Direct Instruction
• Round Table Discussion
• Simulation
• Individual Work

5
Payoff

• Clean data to make ongoing instructional
  decisions that positively impact 3rd grade
  student achievement as evidenced by the WASL

6
Activity One Objective
Develop inter-rater reliability
7
Training Process

• Work in teams of three
• Assign a number (1-3) to each person
• You will need one stack of tests for each group.
• Divide the stack into three piles and give a pile
  to each team member
• Each person will need their own pad of sticky
  notes, copy of rubric and story

8
Step 1

• Read The Cobblers Song silently and answer the
  following test item in writing.
• Compare how the cobbler feels before he receives
  the gold to how he feels after he receives the
  gold. Use information from the story to support
  your answer.

9
Step 2

• Discuss what knowledge, processes and skills are
  necessary to answer this question

10
Step 3

• Review the prompt and accompanying rubric.
  Formulate any clarifying questions you have about
  the item or the rubric.

11
Step 4

• With your team, clarify any questions related to
  prompt or rubric. Are there any adjustments that
  need to be made to the rubric?

12
Step 5

• Score one response 1A as group. Discuss and
  justify why the score was given. Expect
  disagreement at first - work toward consensus
  based on rubric.

13
Step 6

• Without discussion score the rest of the tests in
  your stack
• Record score on post-it note next to the
  response.
• Fold note so score is covered.
• Wait to discuss scores until instructed

14
Step 7

• Team member 1 pass to Team member 2
• Team member 2 pass to Team member 3
• Team member 3 pass to Team member 1

15
Step 8

• Without discussion or looking at the score the
  previous person gave, score each response. Record
  on the outside of the folded sticky note.

16
Step 9

• Continue scoring and passing until all responses
  have two scores.
• Remember, no peeking until all papers are scored!

17
Step 10

• Check the two scores for agreement.
• If there is not agreement on a score, have a
  third person score the item.
• Each person in the group should justify how they
  scored the item.
• Come to group consensus on score.

18
Step 11

• As a large group, review consensus score. Discuss
  any discrepancies and come to consensus.

19
Step 12

• Repeat steps 1-11 for the 2nd item, and then
  repeat again for remaining items.

20
Activity 2 - Objective

• Recognize the range of quality for each scoring
  level

21
Step 1

• Sort papers piles by score

22
Step 2

• Within each pile, organize the papers from best
  to worst

23
Debrief

• What did you notice as you were participating in
  this protocol?
• How might this protocol benefit teaching and
  learning at your site?

24
Activity 3 Objective

• Process for selecting anchor papers

25
Anchor Paper

• Examples of performances that serve as a
  standard against other papers or performances may
  be judged often used as examples of performances
  at different levels on a scoring rubric.

26
Activity 3 Step 1

• Choose a middle of the road or representative
  paper for each scoring level.

27
Step 2

• As a group, annotate each anchor by writing a
  detailed description of the criteria or qualities
  the response exhibits that qualifies it for a
  representative sample of the score point.

28
Example of Annotation

• This response received 0 points because it was
  off topic and did not address prompt. It did not
  contain any information about the cobblers
  feelings before or after he received the gold. No
  text-based information related to the prompt was
  provided.

29
Debrief

• Which papers did you select as anchors?
• What was your annotation?
• Why did you select them?

30
Selecting Exemplars - for each question

• Building level activity
• Select several examples of good responses
• Use these as teaching models
• Keep with rubrics to assist with scoring

31
Using Data

• Provide feedback to students
• Design and adjust instruction
• Predict performance on WASL
• Determine grades for report cards

32
Using data to create if/then statements

• Translate to a percentage
• Divide total earned points by total points
  possible
• Exercise caution when translating percentages to
  grades

33
Calculate Percentages

• 4 multiple choice - 1 point each for 4 points
• 2 short answer - 2 points each for 4 points
• 1 extended response - 4 points each for 4 points
• Total points possible 12 points
• Divide earned points by total possible points

34
Example

• Smart Susie got all of the multiple choice items
  correct for 4 points, she received 2s on both of
  the short answer items and 3 out of 4 points on
  the extended response item.
• 11 points earned/12 points total 92

35
Guided Practice - Intervention Ivan

• 3 multiple choice items correct
• 1 point on short answer item 1
• 0 points on short answer item 2
• 1 point on extended response item
• What percentage score would he receive?

36
If/Then Statements

• If a students usually scores above 75, then we
  can predict that he/she will meet standard with a
  high degree of confidence
• If a student usually scores within the 70-75
  range, he/she may meet standard. Hot list these
  students
• If a student usually scores less than 70, then
  they are unlikely to meet standard.
• (assumes student is reading at 3rd grade level or
  above)

37
If/then Example
38
Guiding Questions

• What patterns do you notice? Strengths and
  opportunities?
• What questions does this data generate for you?
• What other data do you need?

39
Student Accountability (Grades)

• Hold students accountable only after
• skills have been taught
• multiple opportunities to practice have been
  given
• feedback has been provided

40
Standards Based Report Cards

• 4 Student demonstrates mastery of 3rd grade
  learning targets/GLEs that were taught and
  practiced and those that were not yet taught or
  practiced or GLEs above 3rd grade level
• 3 Student demonstrates mastery of most 3rd
  grade learning targets/GLEs that have been taught
  and practiced
• 2 Student demonstrate mastery of some 3rd grade
  learning targets/GLEs that have been taught and
  practiced
• 1 Student demonstrates mastery of few 3rd grade
  learning targets/GLEs that have been taught and
  practiced

41
Checklists

• Whole class
• Individual
• Tailor to meet your needs

42
Guiding Questions

• What patterns do you notice? Strengths,
  Opportunities?
• What questions does this data generate for you?
• What other information do you need?

43
Test Summary
44
Further Reading

• Scoring Rubrics in the Classroom Using
  Performance Criteria for Assessing and Improving
  Student Performance (Experts In Assessment
  Series) Judith A. Arter
• Transforming Classroom GradingRobert J. Marzano