New York State Education Department Regents Examination Updates and Test Development Process

1
New York State Education Department Regents
Examination Updates and Test Development Process

• NYSCEA
• May 2008
• Candace Shyer
• Bureau Chief, Test Development

2
Testing Updates

• Integrated Algebra
• www.emsc.nysed.gov/osa/mathre
• Physical Setting/Earth Science Performance test
  (Part D)
• http//www.emsc.nysed.gov/osa/hssci
• Regents Examinations, Regents Competency Tests,
  and Second Language Proficiency Examinations,
  School Administrator's Manual (2008 Edition)
• http//www.emsc.nysed.gov/osa/hsgen

3
Integrated Algebra

• June 2008 answer sheets and scoring materials are
  printed and shipped to schools by late May
• Schools must complete the top portion of the
  answer sheets and all demographic information
  prior to the examination
• Students take the examination on June 17 at 915
  a.m.
• For June 2008, schools will be required to ship
  their scored answer sheets to the contractor
  (Pearson) by the end of the day following the day
  of administration of the examination (6/18)

4
Integrated Algebra (cont.)

• Schools must make a graphing calculator available
  for the exclusive use of each student
• Other materials ruler/straight edge, 2 pencil,
  blue or back pens
• Only Pearson answer sheets can be used for the
  June 2008 administration

5
Integrated Algebra (cont.)

• Standard setting June 23
• SED is seeking teachers from certain race/ethnic
  groups who currently teach Mathematics A and/or
  Integrated Algebra
• IA Measurement Policy Committee (April 17 June
  23)
• Pre-policy meeting - to provide recommendations
  for the ideal percent of NYS students who should
  be classified in each achievement level
• Post-policy meeting - review June 2008
  operational test data and make revised
  recommendations of the ideal percent of students
  that should be classified in each performance
  level on examination
• Recommendations go to the Technical Advisory
  Group and finally to the Commissioner
• Conversion chart to convert the raw score to the
  scale score posted by Rating Day (June 26)

6
Physical Setting/Earth Science Performance Test
(Part D)

• Three stations rather than four
• Administered for the first time in conjunction
  with the June 2008 Regents Examination in
  Physical Setting/Earth Science
• Administered the last 2 weeks of course but no
  later than the day before the written test (6/20)
• Nine minutes at each station
• Part D component continues to contribute 15 to
  each students final examination score

7
Physical Setting/Earth Science Performance Test
(Part D)

• Change in procedures for students retaking the
  Regents Examination
• Effective June 2008 and thereafter
• Scores earned on Part D in conjunction with prior
  administrations of the written examination may no
  longer be carried over and applied to the
  students final score
• The performance test must be taken each time the
  student takes the written examination

8
Regents Examinations, Regents Competency Tests,
and Second Language Proficiency Examinations,
School Administrator's Manual (2008 Edition)

• Section 1 General information on purpose and use
  of examinations, admission requirements, etc.
• Section 2 Requesting and administering
  examinations
• Section 3 Rating papers, recording scores
• Section 4 Returning test materials to SED
• Section 5 Review of examination papers
• Section 6 Granting of credit and Regents
  endorsement of local diploma
• Section 7 Appendices
• Section 8 Index

9
Test Development Process
10
Phase 1 Test Specifications

• What are test specifications?
• Test specifications provide a general picture of
  the structure of the exam

11
Test Specification Meetings

• Teacher committees meet to make recommendations
  to SED as to the number and type of questions for
  the test
• Teacher committees make recommendations to SED as
  to what percentages of the total credits on the
  test should align with each content strand or
  standard

12
Test Specification Meetings

• Integrated Algebra -November 2006
• Geometry and Algebra2/Trigonometry - March 2007
• Earth Science Performance Test
• (Part D) March 2006
• SED approved Test Specifications may be found at
  www.emsc.nysed.gov/osa

13
Test Item Writing

• Vendor contracts to develop the test in
  conjunction with the SED and NYS teachers
• Regents Examination questions are written by NYS
  teachers

14
Item Writing Workshop

• Professional development opportunity - teachers
  receive training on writing questions and scoring
  rubrics
• First step in creating a bank of questions to be
  used on field tests and operational tests
• Item writer orientation - several sessions each
  year/1-2 days
• Item writer application
• http//www.emsc.nysed.gov/osa/itemwriterapp

15
Item Writing Workshop (cont.)

• New York State Teachers write multiple choice and
  constructed-response questions and their scoring
  rubrics
• Items are sent to SED for review by staff and
  consultants
• Items are then further reviewed by teacher review
  committees

16
Item Review Workshop

• Second step in preparing items for field tests
  and operational tests
• Teachers receive training on reviewing items and
  scoring rubrics written by item writers
• Teachers review items and scoring rubrics to make
  sure they align with the assigned content
  performance indicator

17
The Field-Test Phase

• Definition of Field Testing The first round of
  field testing helps to identify and select the
  best questions for the final operational test
  forms and to identify bias in a given test
  question

18
Field Testing (cont.)

• Items developed at the item writing workshop are
  used to assemble field test forms
• Several committees of teachers work with SED
  staff to prepare and review a series of field
  tests
• All possible field test items undergo additional
  content review prior to actual field testing
• A field test sample is selected for all questions
  and students take those field tests. These are
  secure test forms and schools must maintain their
  security. Schools may not make any notes or
  photocopies of the field tests.

19
Field Testing (cont.)

• The field tests are returned, checked in, rubrics
  are refined, and the papers are scored
• Field tests undergo statistical analysis that
  looks at how the items perform individually as
  well as how they perform together as part of a
  test form using both classical and Item Response
  Theory statistical analysis
• Teachers and schools throughout NYS play a vital
  role in making this process work

20
The Importance of Field Testing

• Essential step in developing valid and reliable
  tests for all students in the State
• Ensures that questions are tied to the Standard
  and the Core Curriculum and are fair and
  representative for all students
• Are administered just before the operational
  Regents Examination
• Provide experience for students in responding to
  the types of questions they will answer on the
  Regents Examination

21
Operational Phase

• Based on field test results the operational
  exams are assembled and reviewed
• A rating guide is prepared (by teachers working
  with SED staff) using actual student work from
  field testing
• An exam is presented to a Final Eyes review
  panel of 3-5 teachers from around the State.
  Application http//www.emsc.nysed.gov/osa/finaley
  es/home
• Teachers provide feedback on State examinations
  through the Teacher Evaluation Form
  www.emsc.nysed.gov/osa/exameval

22
Further questions or comments

• E-mail emscreminfo_at_mail.nysed.gov
• OSA Web Site http//www.emsc.nysed.gov/osa