Alaska Presentation

1
A Standard SettingProcess for Writing
Assessments
2
Standard Setting Process Writing Assessments
The Standard Setting process helps to determine
the final recommendations for Cut Scores for each
Performance Level.
3
Standard Setting Process Writing Assessments
There are four Performance Levels1.
Advanced2. Satisfactory3. Limited Knowledge4.
Unsatisfactory The Unsatisfactory level is
defined as all scores below the Composite Cut
Score for the Limited Knowledge level.
4
Standard Setting Process Writing Assessments
A students Performance Level indicates that the
student can perform the majority of what is
described for that level and even more of what is
described for the levels below. The student may
also be capable of performing some of what is
described in the next level but not enough to
have reached that level.
5
Standard Setting Process Writing Assessments

• 18-24 members per panel, per grade.
• Panel members meet for approximately two full
  days at a conference site.
• There is a panelist selection process often
  determined by the Department of Education.
• Note Qualified panelists must have knowledge
  of the subject matter assessed at the
  appropriate level.

6
Standard Setting Process Writing Assessments
Diverse panels are important to the success of
the Standard Setting process. Panels should have
a healthy mix of

• Diverse ethnic backgrounds
• Geographic representation from districts around
  the state
• Both genders
• Teachers who represent special populations such
  as
• English language learners
• Special Education

7
Standard Setting Process Writing Assessments
Panelists engage in a training session prior to
the actual work. Training helps panelists gain a
clear understanding of

• the various activities that take place during the
  Standard Setting process,
• the meaning of Performance Levels and Cut Scores
  and how they impact the writing assessments,
• the approach used to determine Cut Scores, and
• key concepts.

8
Standard Setting Process Writing Assessments
One method used for writing standard setting is a
hybrid of the Dominant Profile method and the
Body of Work method. This methodology allows
panelists to first identify preliminary cut
scores by defining minimum rubric requirements a
second set of cut scores are empirically
determined based on the previous performance
data. Panelist then, examine student papers
spanning the range of the different sets of Cut
Scores to make a final recommendation for the Cut
Score at each Performance Level.
9
Standard Setting Process Writing Assessments

• Panelists review the writing prompt for their
  grade and are asked to develop an outline as to
  how they would answer the prompt from a students
  perspective.
• Panelists then compare their outlines and
  identify similarities and distinct differences.
• A current scoring rubric is provided
• when panelists complete these
• tasks.

10
Standard Setting Process Writing Assessments

• Panelists take a moment to familiarize themselves
  with the Performance Levels before proceeding.
• Each panel discusses what they feel should be the
  minimum requirements for each Performance Level
  and go about developing descriptions of the
  target or borderline student for each level.

11
Standard Setting Process Writing Assessments

• Beginning with the Satisfactory Performance
  level, panelists review the five Analytic Traits
  and determine a minimum acceptable score.
• The same task is repeated for the Advanced and
  Limited Knowledge levels.

12
Standard Setting Process Writing Assessments

• Each panel continues to work as a group to reach
  consensus on the minimum Cut Scores required for
  each Performance Level.

13
Standard Setting Process Writing Assessments

• Care is taken to not force consensus. Panelists
  continue to discuss profiles until they are
  comfortable with the ranges that they define.

14
Standard Setting Process Writing Assessments

• Workshop facilitators present the Cut Scores on
  the current assessments that are equitable to the
  previous years assessment.
• Facilitators explain this methodology to the
  panelists and show the results of the cut scores.
  
• This process is repeated for all of the
  Performance Levels.

15
Standard Setting Process Writing Assessments

• During these stages of the process, panelists
  work together as a group to build consensus.
• Consensus is not forced panelists record their
  recommendations independently.
• Recommendations are sent to the Department of
  Education and or the States Board of Education
  for final approval.