Lessons Learned from Peer Review

1
Lessons Learned from Peer Review

• 2006 National Conference on Large Scale
  Assessment
• Theresa G. Siskind
• South Carolina Department of Education
• tsiskind_at_sde.state.sc.us

2
Context

• South Carolina has a mature 3-8 assessment
  system.
• The Education Accountability Act (1998) mandated
  assessments in 3-8 in ELA, mathematics, science
  and social studies.
• ELA and mathematics were operational in 1999
  science and social studies in 2003.

3
Context

• The new High School Assessment Program in ELA and
  mathematics was operational in 2004.
• PACT-Alt became operational in 2002 HSAP-Alt in
  2004.
• End-of-course tests became operational in 2004
  for algebra 2005 for physical science, biology,
  English 1 2008 for U.S. History.

4
I thought I was ready for peer review

• Technical Documentation
• http//www.myscschools.com/offices/assessment/Publ
  ications/Index_of_Technical_Reports.htm
• Test Administration Manuals
• Users Guide to Score Reports
• http//www.myscschools.com/offices/assessment/Publ
  ications/Manuals2.htm

5
Technical Documentation

• Chapter 1 History and Overview
• Chapter 2 Test Development
• Chapter 3 Test Administration
• Chapter 4 Scoring
• Chapter 5 Technical Characteristics
• Chapter 6 Item Calibration and Scaling

6
Technical Documentation

• Chapter 7 Setting Performance Standards
• Chapter 8 Reliability
• Chapter 9 Validity
• Chapter 10 Score Interpretation
• Subsequent chapters on special studies
• (Used as model at NCME training sessions)

7
Beyond Compliance

• Comprehensive view of assessment system
• Collaboration among Departmental offices
• A different view of diversity
• Impetus for expediting initiatives
• Cyclical review
• Documentation

8
Assessment System

• Section 59-18-310. (A) Notwithstanding any
  other provision of law, the State Board of
  Education, through the Department of Education,
  is required to develop or adopt a statewide
  assessment program to promote student learning
  and to measure student performance on state
  standards and
• (1) identify areas in which students need
  additional support
• (2) indicate the academic achievement for
  schools, districts, and the State
• (3) satisfy federal reporting requirements and
• (4) provide professional development to
  educators.

9
Assessment System

• Section 59-18-310.(A) Notwithstanding any other
  provision of law, the State Board of Education,
  through the Department of Education, is required
  to develop or adopt a statewide assessment
  program to promote student learning and to
  measure student performance on state standards
  and
• (1) identify areas in which students need
  additional support
• (2) indicate the academic achievement for
  schools, districts, and the State
• (3) satisfy federal reporting requirements and
• (4) provide professional development to
  educators.

10
Collaboration

• Districts
• Other Offices
• Relationships
• Collaboration
• Peer Review
• Within Division/Outside Division
• Federal Programs
• Exceptional Children
• Curriculum and Standards
• School Quality
• Research and Accountability

11
Diversity

• Disaggregated reports have a long history in
  South Carolina
• Traditional checks for sensitivity and bias
• Conscientious effort to represent state on
  committees with respect to geography, race and
  gender
• New emphasis on SWD and ELL

12
Impetus for Expediting Initiatives

• Information to inform instruction
• Descriptions of Achievement Levels (DALsPLDs)
• Specific technical analyses
• EAG participation/participation in other studies
• Prioritization

13
Cyclical Review

• Standards and assessments are revised on a cycle
• Tension of trend and revision
• EAA required reviews
• Documentation

14
Considerations for Modifying Technical
Documentation

• Decision consistency analyses
• Hand-scoring statistics
• Test security including monitoring
• Additional studies
• Consequential validity
• Convergent/discriminant validity
• Accommodations
• Comparability
• Subscore reporting
• Alignment?

15
Feedback

• Part of calibration for review process
• Personal communication more helpful than notes
• Technical assistance too little, too late
• Implications for the future
• Specific guidance
• ELL and SWD
• Not so specific guidance
• Cyclical review
• Alternate assessment
• Science