Rubrics: Design

1
Rubrics Design Use

• To insert your company logo on this slide
• From the Insert Menu
• Select Picture
• Locate your logo file
• Click OK
• To resize the logo
• Click anywhere inside the logo. The boxes that
  appear outside the logo are known as resize
  handles.
• Use these to resize the object.
• If you hold down the shift key before using the
  resize handles, you will maintain the proportions
  of the object you wish to resize.

• Rebecca Langrall, Ed. D.
• Faculty Development Workshop
• Wichita State University
• March 19, 2004

2
Overview

• Prior Knowledge Questions
• Defining Attributes
• Exercises
• Break out Sessions
• Refining existing rubrics
• Developing new rubrics based on targeted outcomes

3
First Thoughts

• Write for a minute about your
• ? ?s
• ? Comments
• ? Concerns
• about rubrics

4
Prior Knowledge

• Turn to someone near you and take turns
  discussing what each of you wrote
• Identify themes among the people at your table.

5
Possible Questions

• Why use them?
• What are rubrics?
• When are they useful?
• How do you design them use
• them reliably?

6
Why Use Rubrics?

• Capture conceptual understanding
• application of knowledge

• For all stakeholders, can clarify
• expectations and degrees of
• outcome mastery

Empower students to become more
independent self- reflective
7
What are Rubrics?

• Originally First letter
• in an Illumination

Now A systematic scoring
methodology to make qualitative
assessment evaluation more reliable
and objective by applying
pre-determined criteria

• Names facets of the targeted performance
• Describes quality levels for each aspect

8
Two Types
1. Holistic A complex performance is
evaluated scored as a whole.

• E.g. Collaborative learning
• http//www.horizon.nmsu.edu/ddl/collaborativerub
  .html
• U.S. History
  http//score.rims.k12.ca.us/history/scoring_guide.
  html

• Pro
• Accommodate overlap-
• ping criteria
• Less time to create than
• an analytic rubric

Con Student may exhibit traits at 2 or
more levels at the same time While
scoring, criteria for each level may be
time consuming to review.
9
Analytic Rubrics
2. Analytic Separate facets of the
performance are defined, and independently
evaluated scored
Example String improvisation development

• Facets Melody ear training
• harmonics rhythm
• bowing backup confidence

10
Excerpt from an Analytic Rubric for String
Improvisation
Quality Levels for Melody --
11
Analytic Example String Improvisation

• Pro
• Sharper focus on target for student, teacher,
  others
• Student is equipped with specific feedback for
  improvement
• Student sees value in work beyond a letter grade

Con Time needed to articulate components of
target performance and find language clear enough
to define performance levels effectively.
12
When To Use Rubrics?

• Yes, for measuring
• Content analysis of open-ended data
• Complex, authentic processes /or
• products, e.g.
• Written or oral narratives
• Diagrams or models
• Behavioral demonstrations

• No, for measuring
• innate attributes impervious to
• instruction. (Popham,1997)
• Close-ended data forced choice response

13
How to Make Rubrics

• Tip 1 Avoid odd of quality levels

• Tendency to dump to the middle to avoid
• making finer judgments

• Four levels provide clearer variations to
• guide student self-assessment

14
Rubric Development Tip 2

• Define poles and work inward
• ? Additive approach -- quantify skills /or
• specify changing conditions

Core I Dispositions Rubric Excerpt

Communicates disagreement/ disapproval of others
statements, states an alternative position, and
provides justification for that position
15
Rubric Development Tip 2

• Define poles and work inward

• ? Error pattern approach, e,g,
• U.S. History Rubric for Research
• Presentation
• http//12.166.242.211/faculty/cwalton/rubric.htm

16
Rubric Development Tip 3

• Unpack abstract language Aim for recognition
• vs interpretation

Professionalism Goal 4 compile maintain
important educational information in order to
share with the school and community.
17
Rubric Development Tip 4

• Involve students in development process

? Offer samples of student work. Have students
use the rubric and associated standards to
critique both the samples the rubric.
Incorporate their feedback.
Results in a high degree of task ownership
understanding of the evaluation criteria
18
Rubric Development Tip 5

• Continua can offer more precision

19
Rubric Development Tip 6

• Rubric development is an iterative process

20
What about Reliability?

• Relates to consistency across time and among
• scorers

• Purpose objectives must guide development.

• Need to develop shared perceptions of nature of
  
• target and what achievement evidence looks
  like.

• Identify anchor samples to accompany
  descriptors.

• Be aware that fatigue and mood can affect
  intra-rater
• reliability

21
Caveats

• A well designed rubric cannot compensate for a
  poorly designed assessment instrument. Moskal
  Leyden, 2000

Faculty as a group must clearly delineate their
specific expectations by level for all rated
features. (Pino, B.G., 1998)
22
Comments Questions
23
References

• Haag, S. Kenimer, A. (2003). Constructing
  rubrics for open-ended activities. ASEE
  Workshops, June 22, 2003. Available on line at
• http//www.eas.asu.edu/assess/ppt/Rubrics20wors
  hoop203-14_2002.ppt
• Pino, B.G. (1998). Prochievement testing of
  speaking Matching instructor expectations,
  learner proficiency level, and task type. Texas
  Papers in Foreign Language Education, 3 (3), 119
  - 133. ED427522
• Moskal, G. M. Leydens, J.A. (2000). Scoring
  rubric development Validity and reliability.
  Practical Assessment, Research Evaluation,
  7(10). Retrieved March 14, 2004 from
  http//PAREonline.net/getvn.asp?v7n10.
• Popham, W.J. (1997). Whats wrong and whats
  right with rubrics. Educational Leadership,
  55(2). Available on line at http//www.ascd.org/
  safeschools/el9710/pophamrubric.html

24
Image Credits

• Errorclassic.mountainzone.com/.../
  photo07_int.html
• Feedback www.wildtech.org/ feedback.htm
• Profesionalism www.pafossearch.com/
  housekeeping.php
• Psalm Page umsis.miami.edu/kwisniew/ misc.htm
• Self-Evaluation www3.sympatico.ca/jim.rowe2/
  p12-8.htm
• Thumbs Down www.cafeshops.com/absenteeshirts/
  130050
• Thumbs Up www.labor.iu.edu/.../Media20Files2
  0Main.htm

25
Contact Information

• Rebecca.Langrall_at_wichita.edu
• Phone (636) 537 - 4665
• Fax (636) 536 - 4629
• Address
• 1838 Elmsford Lane
• Chesterfield, MO 63005