Introduction to Standard Setting

1
Introduction to Standard Setting

• Scott Marion
• Wyoming Department of Education
• December 3, 2001

2
Overview of the day

• Introduction to standard-setting
• Overview of two methods
• Work through examples for two methods
• Q A

3
Introduction

• Our task is to use a deliberative process to
  identify points along a continuum to classify
  student performance into multiple categories
• Standard setting is controversial, but it is
  almost always better than using traditional
  percentages (e.g., 80)
• Safest road is to ensure certain fundamental
  procedures are followed
• Documenting your approach will help you in the
  long run (and even the short run)

4
Important Distinction

• Standard-setting panels do not set standards
  they are at least one form of input, albeit an
  important one, into the standard-setting process.
• Your Board or a group sanctioned by your Board,
  using the evidence it sees fit, sets the
  standards. It will be evaluated on whether it
  used appropriate input in a reasonable fashion.

5
It all starts with the performance descriptors

• Standard setting is the process of putting
  specific meaning on these general statements.
• If the performance descriptors change, you have
  to restart the process, because the descriptors
  have a great deal of influence on where the
  cutscore score are established.
• If you make general statements more specific, you
  constrain the standard-setting panels.

6
Performance descriptors

• In Wyoming, we need to generate performance
  descriptors based on the performance standards in
  the Wyoming Content and Performance Standards
  (you can just adopt the state standards).
• The descriptors need to be written for either the
  standard level or the content area level (the
  minimum requirement).

7
Classes of Standard-Setting Procedures

• Focus on
• Tasks/Items (Angoff, Bookmark)
• Examinees (Contrasting Groups, Teacher Judgment,
  Borderline)
• Examinee responses (Body of Work)
• Scored performances (Profiles)
• All of these were designed to set standards on
  single testsmost are not adaptable for a
  complete BOE system.

8
Standard Setting Methods You Are Most Likely to
Use

• Body of Work
• Contrasting Groups/Teacher Judgment
• Borderline Groups

9
Focus on Examinee ResponsesBody of Work

• Select work from a range of students
• Have panel determine which students are above
  standard and which are below, on the basis of the
  work shown
• More details later

10
Combining judgments

• Ultimate goal is a pass/fail decision on each
  student in each content area
• Separate standard-setting process for each
  content area, but it might be easier to set
  standards for each content standard and then
  integrate judgments
• Can combine parts through Body of Work or profile
  methods
• Standards across content areas should be
  consistent at least according to expectations of
  performance

11
Focus on ExamineesContrasting Groups

• Administer assessments
• Identify proficient and non-proficient
• Select score that puts most proficient students
  and fewest non-proficient students above that
  score

12
Focus on ExamineesContrasting Groups
Non-Proficient
Proficient
Cut Score
13
Dilemma with Contrasting Groups
Masters
Non-Masters
Cut Score
14
Dilemma with Contrasting Groups
Percentage of Non-Masters
Masters
Number of Non-Masters
Where Should the Cut Score Go?
15
Focus on ExamineesTeacher Judgment Method

• Collect judgments of performance for a sample of
  students-tied closely to performance descriptors
• Associate judgments with scores on set of
  assessments
• Select score that puts most students in upper
  category and fewest students in lower category
  above that scorethe same procedure as
  contrasting groups

16
Focus on ExamineesBorderline Group Method

• Administer assessments
• Identify borderline candidates
• Select an average score for borderline candidates
• Place cutscore

17
Focus on ExamineesBorderline Group
Borderline Candidates
18
Focus on ExamineesBorderline Group
Borderline Candidates
Cut Score
19
Selecting Panelists

• Number
• Composition
• Stakeholders (i.e., your Board)
• Experts
• Public
• Represented constituencies
• Process for selecting each panelist
• Motivation for participating
• Sub-panels
• Homogeneous vs. heterogeneous
• Qualification decisions

20
Training

• Background, purpose
• Definitions
• Content standards
• Performance standards
• Distinction between panelist and policy-maker
• Practice with authentic materials, i.e., student
  work

21
Options to Choose

• Composition of panel
• Size
• Experts vs. stakeholders
• Homogeneous vs. designed heterogeneity
• Training of panel
• Focus on gaining an operational understanding of
  the descriptors
• Calibration of panelists to develop consistency
  in ratings

22
Options to Choose (contd)

• Degree of independence of panelists judgments
• Number of iterations
• Amount of material to show panelists
• Timing and amount of feedback
• Summarization procedures

23
Checking Validity

• Consistency across content areas
• Consistency with qualified judgments
• Consistency with external measures, e.g.,
  national or state tests

24
Checking Reliability

• Numbers of panelists
• Different samples of student work
• Different standard-setting procedures
• Different types of panelists
• Different definitions of mastery
• Different feedback provided

25
Documentation

• This will be important for
• Review and feedback
• Potential legal challenges
• Consistency of program in cases of personnel
  turnover