Integration of Existing High School GraduationType Tests into NCLBCompliant Tests: Challenges and In

1
Integration of Existing High School
Graduation-Type Tests into NCLB-Compliant Tests
Challenges and Innovative Solutions

• 36th Annual National Conference on
• Large-Scale Assessment
• San Francisco June 28, 2006

2
Session Purpose

• Legislated Standards
• NCLB mandated by Feds
• One administration
• Typically given in spring
• One grade in High School (10 12)
• High School Graduation Examination (HSGE)
  mandated by States
• Usually already in place
• Has multiple opportunities for students to take

3
Session Purpose

• Scenario
• Two sets of standards
• High Standards for NCLB
• Competency Standards for HSGE
• Two separate assessments
• Two separate score reports

4
Session Purpose

• Scenario Continued
• Generally higher test reliability for both tests
• More remediation/diagnostic information
• Increased administration costs
• Student motivation issues
• Lose more instructional time
• Redundant item level information

5
Session Purpose

• Use one assessment
• or
• two assessments?

6
Session Purpose

• This session will share a variety of challenges
  and innovations in using one assessment to
  provide both graduation and accountability
  information.

7
Session Presenters
8
Presenter

• Dr. Chris Domaleski
• Georgia Department of Education

9
Georgia High School Graduation Tests (GHSGT)

• State law requires that curriculum-based
  assessments be administered in Grade 11 for
  graduation purposes.
• Phased in starting in 1996

10
Georgia High School Graduation Tests (GHSGT)

• Results are used to identify students who may
  need additional instruction in content area
  objectives considered essential to earn a high
  school diploma in Georgia.
• Assessments cover a sample of the knowledge and
  skills that constitute a comprehensive high
  school education.

11
GHSGT Content Areas

• English/Language Arts
• Reading/Literature
• Critical Thinking
• Writing, Usage, Grammar
• Mathematics
• Number and Computation
• Data Analysis
• Measurement and Geometry
• Algebra

12
GHSGT Content Areas

• Science
• Process/Research/Safety skills
• Physical Science
• Biology
• Social Studies
• World Studies
• U. S. History to 1865 and since 1865
• Citizenship/Government
• Map Interpretation Skills
• Information Processing Skills

13
GHSGT Administration

• Students assessed spring of 11th grade year
• Students have 5 opportunities to test prior to
  graduation
• Administrations September, November, March
  (Main), and July

14
GHSGT Enhanced

• What prompted to the revision?
• Federal standards call for increased rigor in
  defining proficiency at the school and system
  level.
• Desire to maintain the longitudinal comparability
  of scores and to keep the bar at the same level
  to determine diploma eligibility

15
GHSGT Enhanced

• Involved two primary changes
• Addition of more cognitively complex items
• Establishment of new performance levels

16
GHSGT Enhanced

• Blueprint minimally impacted
• Additional items selected from existing bank
• New performance level descriptors developed for
  Enhanced performance levels
• Modified-Angoff standard setting procedure
  conducted in Spring 2004

17
GHSGT Enhanced

• Language Arts 60 items
• Subtracted 2 items from the original 50 item
  test
• Add 7 enhanced items
• Add 5 field test items

18
GHSGT Enhanced

• Mathematics 70 items
• Subtracted 2 items from the original 60 item
  test
• Add 7 enhanced items
• Add 5 field test items

19
E-GHSGT Language Arts Scale
500Pass
538Pass Plus
400
600
538Advanced
511Proficient
20
E-GHSGT Mathematics Scale
500Pass
535Pass Plus
400
600
525Advanced
516Proficient
21
GHSGT - Enhanced

• Pass and Pass Plus standards are used for student
  accountability.
• Proficient and Advanced standards used for school
  level AYP determinations.

22
GHSGT - Enhanced

• Performance standard for diploma eligibility has
  not changed.
• Each test is evaluated twice -once for the
  enhanced set of items and once for the base set.
  A student who scores 500 or better on either set
  will pass.

23
Presenter

• Mr. Les Morse
• Alaska Department of Education
• and
• Early Development

24
Alaska High School Test

• Alaska State Statute 14.03.075
• A student may not be issued a secondary school
  diploma unless the student passes a competency
  examination in the areas of reading, English, and
  mathematics or receives a waiver from the
  governing body.
• The intent language indicates the test is
  essential skills of minimum competency.

25
Alaska High School Test

• Federal Law - No Child Left Behind
• Assessment of student performance relative to the
  state standards.
• No out of grade level testing

26
Alaska High School Test

• High School Graduation Qualifying Examination
  (HSGQE)
• Required since 2001
• Standards (passing proficiency levels)
  established in 2000 and 2002
• Recommending changes to standards in 2006
• First class required to meet requirement was 2004

27
Alaska High School Test

• Impact of NCLB on Exam
• HSGQE did not test grade level standards
• HSGQE was re-focused in 2001/2002 to concentrate
  on essential skills of minimum competency
• Use of HSGQE would require changing state law, or
  adjust the test to meet both purposes

28
Alaska High School Test

• SBA / HSGQE Integrated Exam Timeline
• Technical state expert feedback fall 2004
• TAC meetings fall / winter / spring 2004/05
• State board gives approval March 2005
• Design and item development - 2004/05
• Field test items of both exams 2005 (continues)

29
Alaska High School Test

• SBA / HSGQE Integrated Exam Timeline cont.
• Students take exam for first time April 2006
• Students in grades 11 and 12 (spring) and
  students retaking (fall) - only HSGQE component
  of exam
• Standard setting May 2006
• Individual student reports 2 different results
  reported

30
Example Mathematics Blueprint
31
Example Reading Blueprint
32
Example Writing Blueprint
33
Alaska High School Test

• SBA / HSGQE Integrated Exam
• Meets requirements under state and federal
  statute
• Eliminates the need for an additional test to
  meet NCLB requirements, or need to change state
  statute
• No loss of instructional time

34
Presenter

• Dr. Donna Butterbaugh
• Data Recognition Corporation

35
Psychometric Challenges

• How?

36
Psychometric Challenges Cont.

• Philosophical Issues Pertaining to Assessments
• Competency
• Content based on essential skills and often
  involve standards from lower grade levels
• Focus is on what a student can do
• On grade level
• Content based on High Standards
• Focus is on what a student should be able to do

37
Psychometric Challenges Cont.

• Item Development
• Items are written so that they are aligned to
  standards
• It is possible to align some items to both sets
  of standards
• Some items may be HSGE-only while others are
  NCLB-only

38
Psychometric Challenges Cont.

• Field Testing
• Embedded or appended field tests in the spring
  administration
• Poses issues for what to do for the retest
  population.
• Administer all field test items to retesters
• Administer HSGE-only field test items to
  retesters
• No field test items administered to retesters

39
Psychometric Challenges Cont.

• Test Design
• What to do with retesters?
• Administer field test items?
• Administer NCLB operational items?
• Separate retest form
• What happens if a retester takes the wrong form?
• What happens if a NCLB AYP student takes the
  retest form that doesnt contain NCLB items?

40
Psychometric Challenges Cont.

• Test Design continued
• When two sets of standards are involved in a
  dual-purpose test which standards are emphasized?
• High Stakes for Students (Diploma)?
• High Stakes for States/Districts/Schools/Teachers?
  
• Equally emphasized?
• Equally reliable?

41
Psychometric Challenges Cont.

• Test Design continued
• How does test design effect the reliability?
• Need to verify content coverage for each part of
  the assessment as well as the overall test
  document.

42
Psychometric Challenges Cont.

• Test and Item Calibration
• Which students are included in the calibration
  sample?
• Separate calibrations lead to different origins
  and therefore different logit difficulties for
  dual-purpose items.

43
Psychometric Challenges Cont.

• Item Banking Dual-purpose Items
• Separate test calibrations
• If retesters are not administered the field test
  items then the p-values would be the same for the
  calibrations
• Origins would differ in the calibration and
  therefore items would have two logit difficulties
• Separate item banks maintained

44
Psychometric Challenges Cont.

• Scale Score Tables
• Two Calibrations ? Two Equations ? Two Scale
  Score Tables
• Ideally, scale as different as possible to avoid
  confusion

45
Psychometric Challenges Cont.

• Cut Scores
• Typically one cutpoint for HSGE
• This doesnt comply with NCLBs four category
  requirements
• NCLB Standard Setting
• New cutpoints (3) determined
• Add two additional cutpoints

46
Psychometric Challenges Cont.

• Decision Rules
• Grant a diploma based solely on the HSGE
  performance?
• What if student does not pass the HSGE but is
  proficient on the NCLB test?
• Rules vary by State and content area
• Typically large content overlap in
  Reading/Language Arts.

47
Psychometric Challenges Cont.

• Score Reports
• HSGE reports are generally used as guides for
  student remediation
• NCLB reports are generally used for District
  and/or School curricula decisions
• Separate reports versus a single report
• Best to have separate reports to avoid confusion
• Single report possible if different scales are
  used

48
Psychometric Challenges Cont.

• The Georgia Solution
• One set of standards
• High Standards for HS and NCLB
• Georgia High School Graduation Tests (GHSGT)
• Single, dual-purpose, spring assessment
• Fall, winter, spring and summer GHSGT retest
  opportunities

49
Psychometric Challenges Cont.

• Georgia Test Design
• The Enhanced-GHSGT retains the same item emphasis
  as the GHSGT
• The enhanced items are distributed throughout the
  test

50
Psychometric Challenges Cont.

• Georgia Test Design continued
• Retest population had a separate form created for
  the spring 2006 ELA and Science content areas
• In the future retesters will take the same
  enhanced GHSGT test as first time test takers in
  all four subjects

51
Psychometric Challenges Cont.

• Georgia Item Calibration
• Both tests maintain the baseline common scale.

52
Psychometric Challenges Cont.

• Georgia Item Banking
• One item bank maintained

53
Psychometric Challenges Cont.

• Georgia Scale Score Tables
• Two separate scale score tables for the initial
  analysis
• GHSGT cutscore of 500 retained on the enhanced
  GHSGT scale score table
• NCLB cutscores vary by content area.

54
Psychometric Challenges Cont.

• Georgia Score Reporting
• One score report generated

55
Psychometric Challenges Cont.

• Georgia Summary
• Minimal impact on base test reliability
• Original base test length decreased by 2 items
• Maintains the former level of remediation
  information
• Decreased administration costs
• Greater student motivation
• More instructional time

56
Psychometric Challenges Cont.

• The Alaska Solution
• Two sets of standards
• High Standards for NCLB
• Standard Based Assessments - grade 10 (SBA10)
• Essential Skills Standards for HSGE
• High School Graduation Qualifying Examination
  (HSGQE)
• Single, dual-purpose, spring assessment
• Fall and spring HSGQE-only retest opportunities

57
Psychometric Challenges Cont.

• Alaska Test Design
• Attempted to retain historical item emphasis for
  the HSGQE with the use of dual-coded items as
  much as possible
• Success varied by content area
• Mathematics is a shorter test
• Retest population provided a separate form that
  did not include field test or SBA10 items (Form
  15)

58
Psychometric Challenges Cont.

• Alaska Test Design continued
• HSGQE-only and SBA-only items are distributed
  throughout the test
• Students do not know which items count towards
  which test.

59
Psychometric Challenges Cont.

• Alaska Item Calibration
• HSGQE
• Pre-equated, operational HSGQE item difficulties
• Use the entire HSGQE testing population (Forms 1
  15)
• SBA10
• New origin for the operational SBA10 item
  difficulties
• Use only valid grade 10 students that took Forms
  1-14

60
Psychometric Challenges Cont.

• Alaska Item Banking
• Two Item Banks for statistical information
• Because retesters werent administered the field
  test items, the p-values and proportions choosing
  each option will be the same for these items
• Item-Total correlations will differ

61
Psychometric Challenges Cont.

• Alaska Scale Score Tables
• Retain the existing HSGQE cutscores
• Passing cutscores vary by subject
• Scale score range limited to the 100 600 range
• Extend the current grades 3 9 SBA results to
  include grade 10
• Proficient cutscores set at 300
• Scale score range limited to the 100 600 range

62
Psychometric Challenges Cont.

• Alaska Score Reporting
• Two separate reports are generated

63
Psychometric Challenges Cont.

• Alaska Summary
• Attempt to minimize impact on test reliability
• Make the test as long as possible without
  creating fatigue

64
Psychometric Challenges Cont.

• Alaska Summary continued
• Maintain the level of remediation information but
  possibly lose some diagnostic information for
  certain content areas
• Decreased administration costs
• Greater student motivation
• More instructional time

65
Discussant

• Dr. Brian Gong
• Center for Assessment