Peer Review Process

1
Peer Review Process Alignment Study Joe
Willhoft Assistant Superintendent of Assessment
and Student Information Yoonsun Lee Director of
Assessment and Psychometrics Office of
Superintendent of Public Instruction
2
Overview

• Peer Review
• Purpose of Peer Review
• NCLB Peer Review Components
• WA Peer Review Results
• Alignment Study
• Alignment Method
• Alignment Study Example

3
NCLB Accountability Requirements

• NCLB requires states to
• Establish challenging standards
• State must apply the same academic standards to
  all public schools and public school students
• Standards include description of the knowledge,
  skills, and levels of achievement expected of all
  students
• Include at least mathematics, reading or language
  arts, and science
• Develop aligned assessments
• Build accountability systems for districts and
  schools based on educational results

4
NCLB Peer Review

• Each states standards and accountability program
  is subject to review and must be approved by a
  panel of peers
• Panels consist of three assessment professionals
• States submit documentation that the state has
  met the requirements for each of seven critical
  elements
• Categories of approval Not Approved, Approval
  Pending, Approval Expected, Fully Approved (with
  or without recommendations)

5
NCLB Peer Review

• Review August 6, 2008 Letter

6
Seven Peer Review Critical Elements

1. Challenging academic content standards
2. Challenging academic achievement standards
3. System of annual high-quality assessments
4. System of assessments with high technical quality
5. Alignment of academic content standards, academic
   achievement standards, and assessments
6. Inclusion of all students in the assessment
   system
7. An effective system of assessment reports

7
Seven Peer Review Critical Elements

• Each table discuss two elements
• Table 1 Discuss Elements 1 2
• Table 2 Review Elements 2 3
• Table 3 Review Elements 3 4, etc.
• Discussion
• 1. What does this element mean?
• 2. How does Washington addresses this element?
• 3. What impact does this element have on schools?
• Be prepared to report out

8
Table Discussion

• Challenging academic content standards
• Challenging academic achievement standards
• System of annual high-quality assessments
• System of assessments with high technical quality
• Alignment of academic content standards, academic
  achievement standards, and assessments

• Inclusion of all students in the assessment
  system
• An effective system of assessment reports
• What does this element mean?
• How does Washington addresses this element?
• What impact does this element have on schools?

9
NCLB Peer Review

• Review May 5, 2006 Letter

10
1. Challenging Academic Content Standards

• States must develop a set of challenging academic
  content standards. Standards must
• develop grade specific expectations in addition
  to its standards
• define the knowledge and skills that are expected
  of all students prior to graduation (high school
  level)
• be rigorous and encourage the teaching of
  advanced skills
• Standards review by external panel
• Careful review of the grade level expectations
  development process with curriculum and
  assessment personnel
• Online survey to gather feedback on refinements
  to the standards

11
2. Challenging Academic Achievement Standards

• Academic achievement standards must
• include at least three achievement levels (e.g.,
  basic, proficient, and advanced). Proficient and
  advanced must represent high achievement and
  basic must represent achievement that is not yet
  proficient.
• Include descriptions of the content-based
  competencies associated with each level.
  Cutscores must be established through a process
  that involves both expert judgments and
  consideration of assessment results.
• Be aligned with the states academic content
  standards in that they capture the full range and
  depth of knowledge and skills defined in the
  states academic content standards.

12
3. Alternate Academic Achievement Standards

• A state is permitted to define alternate
  achievement standards to evaluate the achievement
  of students with the most significant cognitive
  disabilities. Alternate academic achievement
  standards must
• Be aligned with the states academic content
  standards for the grade in which the student is
  enrolled.
• Be challenging for eligible students, but may be
  less difficult than the grade level academic
  achievement standards
• Include at least three achievement levels
• Be developed through a documented and validated
  standards setting process that includes broad
  stakeholder input.

13
4. System of Annual High-Quality Assessments

• NCLB requires states to develop a single
  statewide system of high quality assessments. All
  public school students must participate in this
  assessment system including those with
  disabilities and those who are not yet proficient
  in English.
• Reading and Mathematics components of the
  assessment systems in place by the 2005-2006
  school year (science by 2007-2008) and must be
  administered annually to all students in each of
  grades 3-8 and at least once to students in the
  10-12 grade range.

14
4. System of Annual High-Quality Assessments
(continued)

• The States assessment system should involve
  multiple measures that assess higher-order
  thinking skills and understanding of challenging
  content.
• WASL includes multiple measures (multiple choice,
  short answer, and extended response items) to
  assess higher order thinking skills and different
  levels of cognitive complexity.

15
System of Assessments with High Technical Quality

• The Standards for Educational and Psychological
  Testing delineates the characteristics of
  high-quality assessments and describes the
  processes that a state can employ to ensure that
  its assessments and use of results are
  appropriate, credible, and technically
  defensible.
• Validity
• Reliability
• Other dimensions of technical quality

16
System of Assessments with High Technical Quality

• Validity Whether the State has evidence that the
  assessment results can be interpreted in a manner
  consistent with their intended purpose(s).
• Evidence based on test content
• Evidence based on the assessments relation to
  other variables
• Evidence based on student response processes
• Evidence from internal structure

17
Validity

• Evidence based on test content (Content
  validity) alignment of the standards and the
  assessment
• Content validity is important but not sufficient.
  States must document not only the surface aspects
  of validity illustrated by a good content match
  but also the more substantive aspects of validity
  that clarify the real meaning of a score
• For WASL, content validity is confirmed by
  content specialists (teachers, curriculum and
  assessment specialists) by examining if each item
  is aligned with content standards.

18
Validity

• Evidence based on the assessments relation to
  other variables Demonstrate the validity of an
  assessment by confirming its positive
  relationship with other assessments or evidence
  that is known or assumed to be valid.
• If students who do well on the assessment in
  question also do well on some trusted assessment
  or rating such as teachers judgments.

19
System of Assessments with High Technical Quality

• Evidence based on student response processes
  Eliminate sources of test invalidity during the
  test development process through reviews for
  ambiguity, irrelevant clues, and inaccuracy.

20
Validity

• Evidence based on internal structure Use
  statistical techniques to study the structure of
  a test.
• Item correlations
• Generalizability analyses
• Factor analysis

21
Reliability

• Reliability is defined with consistency,
  stability and accuracy.
• States assessment systems are obliged to
• Make a reasonable effort to determine the types
  of error that may distort interpretations of the
  findings
• Estimate their magnitude
• Make every possible effort to alert the users to
  this lack of certainty

22
Reliability

• Traditional methods of portraying the consistency
  of test results are
• Reliability coefficients
• Standard errors of measurement
• Actual level of accuracy
• Actual level of consistency

23
Other Dimensions of Technical Quality

• Fairness/Accessibility
• Do the items and tasks provide an equal
  opportunity for all students to fully demonstrate
  their knowledge and skills?
• Are the assessments administered in ways that
  ensure fairness?
• Are the results reported in ways that ensure
  fairness?
• Are the results interpreted or used in ways that
  leads to equal treatment?

24
Other Dimensions of Technical Quality

• Comparability of results
• Comparability from year to year, from student to
  student, and from school to school
• Procedures for test administration, scoring, data
  analysis, and reporting
• Are the assessments properly administered?
• Are directions followed?
• Are test security requirements clearly specified
  and followed?

25
Other Dimensions of Technical Quality

• Interpretation and use of results
• Do the results reflect the goals of instruction,
  especially those related to higher-order thinking
  and understanding?
• Use of accommodations
• Are appropriate accommodations available to
  students with disabilities and students covered
  by Section 504?
• Are appropriate accommodations available to
  limited English proficient students ?
• Do scores for those students (disabilities,
  limited English proficient) allow for valid
  inferences about students knowledge and skills
  and can be combined meaning fully with sores from
  none-accommodated administration circumstances?

26
5. Alignment of academic content standards,
achievement standards, and assessments

• Do a States assessments adequately measure the
  knowledge and skills specified in the States
  academic content standards?
• Do the assessments cover the full range of
  content specified in the States academic content
  standards?
• Do the assessments measure both the content and
  the process aspects of the academic content
  standards?
• Do the assessments reflect the full range of
  cognitive complexity and level of difficulty of
  the concepts and processes descried in the
  States academic content standards?

27
5. Alignment of academic content standards,
achievement standards, and assessments

• Alignment studies should
• Demonstrate the breath and depth of the match
  between assessments and content standards.
• Demonstrate that the performance descriptors are
  consistent with the demands of the test content
  and content standards.
• Document the link between alternate assessments
  based on alternate achievement standards and
  grade level content standards.

28
6. Inclusion of All Students in the Assessment
System

• Inclusion of all students in a States system of
  standards, assessments, and accountability
• For students with disabilities and for students
  who are not yet proficient in English,
  participation in the States assessment system
  may require special considerations.
• For LEP students who have been in school in the
  U.S. for less than 12 months, regulations permit
  the State to substitute participation in the
  Sates English proficiency test for participation
  in the grade level reading test for one year only

29
7. An effective system of assessment reports

• Do a parent, educator, or other stakeholder find
  answers to questions about how well a student or
  group of students is achieving, as well as
  important information on how to improve
  achievement in the future?
• Do States produce reports at the individual
  student, school, LEA, and State levels?
• Reports must include scores that are aligned with
  the Sates academic content standards.

30
Peer Review Process

• To determine whether or not states have met NCLB
  standards and assessments requirements, the U.S.
  Dept. of Education use a peer review process
  involving experts (peer reviewers) in the fields
  of standards and assessments.
• Peer reviewers examine characteristics of a
  States assessment system that will be used to
  hold schools and school districts accountable
  under NCLB.
• Peer reviewers advise the Dept. of Education on
  whether a State assessment system meets a
  particular requirement based on totality of
  evidence submitted.

31
WA Peer Review Results

• In August, 2008 Washingtons standards and
  assessment system were approved.
• The decision was based on input from peer
  reviewers external to the U.S. Dept of Education
  who reviewed the evidence demonstrating that
  Washingtons system includes academic content and
  student academic achievement standards in
  reading, mathematics, and science alternate
  academic achievement standards for students with
  the most significant cognitive disabilities in
  those subjects.

32

• WA Alignment Study

33
Science Alignment Study

• How does the WASL align with Essential Academic
  Learning Requirements (EALRs) and Grade Level
  Expectations (GLEs) in science at the 5th, 8th,
  and 10th grade levels?
• Panels of educators participated in this study.
• The primary task was to evaluate how well score
  points from the WASL science assessments matched
  the state GLEs in terms of content and cognitive
  complexity

34
Methodology

• Frisbie (2003) Webb (1997)
• Judgments
• Cognitive complexity of GLEs and EOLs
• Cognitive complexity of score points for each
  item
• Content fit of score points with GLEs and EOLs
• Scoring guides and exemplar responses for score
  points were available for review by panelists
• Two-stage procedure
• Independent judgment
• Group consensus discussion and recommendation

35
Procedures

• 14 science educators participated in the
  alignment study based on geographic information
  and school size.
• The participants had content and assessment
  expertise.
• The panelist review was facilitated by an
  independent contractor, who also wrote the
  summary report

36
Procedures

• Panelists were asked to rate the cognitive
  complexity for each GLE and Evidence of Learning
  (EOL). Three levels of complexity were used
• Level1 - Conceptual understanding and
  comprehension Assessment items, GLEs, or EOLs at
  this level focus on remembering facts,
  comprehension of concepts, recognizing attributes
  of a process and understanding ideas. Assessment
  items at this level might ask examinees to
  identify, recognize, recall, classify, summarize,
  or compare.

37
Procedures

• Level 2 Application, analysis, synthesis, and
  evaluation Assessment items, GLEs, or EOLs at
  this level focus on application of concepts and
  ideas to human problems and situations through
  predictive analysis, synthesis of information and
  evaluation of situations or problems. Assessment
  items at this level might ask examinees to
  conclude, plan, differentiate, critique, create
  new ideas or meaning, design, explain, evaluate,
  or organize.
• Unclassifiable This level applies when a GLE or
  EOL is worded so ambiguously that it is not
  possible to determine how students are expected
  to interact with the content.

38
Procedures

• After rating the cognitive complexity for each
  GLE and Evidence of Learning (EOL), panelists
  evaluated the degree of fit between item (or
  score point for CR items) and the EOL the item is
  designed to assess. Three levels of fit were
  used
• C Complete fit the main content required to
  answer the item correctly is contained in the
  GLE/EOL. If the student gets the item right, this
  is one relevant piece of information about the
  students level of achievement of the content
  stated in the GLE/EOL

39
Procedures

• P - Partial fit A significant portion of the
  content required to answer the item correctly is
  embodies in the GLE/EOL. But there is additional,
  significant understanding required that is
  represented by some other GLE/EOL. If the student
  gets the item (point) right, it is because the
  student has some other significant knowledge that
  is not part of this GLE/EOL
• S - Slight fit There is some relationship
  between the item content and the content of the
  EOL, but much more is needed to answer the item
  correctly. Alignment would probably be more
  complete with some other GLE/EOL, or it might
  take several GLE/EOLs to cover the content of the
  item sufficiently.
• X - No fit the item does not fit the content of
  any GLE/EOL

40
2007 Results Grade 5

• Overall
• All score points judged to align to one GLE
• Coverage is balanced across content and cognitive
  levels
• Systems of Science
• 20 score points
• Inquiry in Science
• 20 score points
• Application of Science
• 9 score points

41
2007 Results Grade 8

• Overall
• All score points judged to align to at least one
  GLE
• Coverage is balanced across content and cognitive
  levels
• Systems of Science
• 27 score points
• Inquiry in Science
• 24 score points
• Application of Science
• 12 score points

42
2007 Results Grade 10

• Overall
• All score points judged to align to at least one
  GLE
• Coverage is balanced across content and cognitive
  levels
• Systems of Science
• 31 score points
• Inquiry in Science
• 27 score points
• Application of Science
• 9 score points

43
Conclusions

• Results suggest increasingly challenging content
  standards across grade levels.
• Score points were balanced across GLEs on content
  and cognitive complexity.
• Panelists evaluation ratings and comments were
  positive.