Comparing State Reading and Math Performance Standards Using NAEP

1
Comparing State Reading and Math Performance
Standards Using NAEP

• Don McLaughlin
• Victor Bandeira de Mello
• National Conference on Large-Scale Assessment
• San Antonio
• June 2003

2
Another look at NAEPs role in the context of
NCLB

• Confirm state gains.
• Help state assessment programs to help each
  states schools and students.
• Answer questions for/about the state assessment.
• Provide a test bed for new solutions.

3
Complexities in confirming state gains

• Sampling
• Exclusions, Absences, Small samples
• Measurement
• Accommodations, Framework, Format, Time of year,
  Motivational context
• Differential sensitivity
• Analysis
• Gaps
• Design
• Grade-to-grade growth

4
NAEPs strengths for helping state assessment
programs

• NAEP is administered with uniform procedures to a
  representative sample of public school students
  in every state. This can illuminate
  cross-assessment comparisons.
• NAEP can carry out validity research to address
  questions relevant to all assessments (e.g.,
  validity of accommodations).

5
Questions NAEP can answer for state assessments

1. How do your states performance standards for
   math and reading compare with other states?
2. How do your states assessment instruments
   compare to others in focus and difficulty?
3. Can your state assessments identification of
   schools that need improvement be enhanced?

6
Questions NAEP can answer for state assessments

1. Is your states assessment more (or less)
   sensitive to race or poverty differences in
   achievement than other states?
2. Are differences between types of schools in your
   state similar to analogous differences in other
   states?
3. How do your identification and testing procedures
   for SD/ELL students differ from those in other
   states?

7
Comparison of state performance standards, using
NAEP

• Achievement levels are set separately in each
  state, using a variety of standard-setting
  methodologies.
• Does meeting the standard mean something
  different in each state? How different?
• How do states targets for adequate yearly
  progress differ?

8
Comparison of state performance standards, using
NAEP

• By comparing NAEP results with state assessment
  results in the same schools, we can approximate
  where on the NAEP scale the cutpoint for the
  states standard lies.
• This requires matching a database of state
  assessment results to the NAEP database.

9
National Longitudinal School-Level State
Assessment Score Database

• Scores provided by
• State Education
  Agencies
• Processed by
• the American Institutes for
  Research
• Funded by
• the U.S. Department of
  Education
• Policy and Program Studies Service
• School-level scores in the public domain.

10
National Longitudinal School-Level State
Assessment Score Database

• Scores for gt 80,000 public schools
• in 49 states, Puerto Rico, and DC
• from 1997-1998 through 2001-2002
• (but not all years in all states)
• merged with Common Core of Data

11
National Longitudinal School-Level State
Assessment Score Database

• Available at
• www.schooldata.org
• www.edtrust.org
• www.greatschools.net
• Contact statedata_at_air.org

12
(No Transcript)
13
Method

• Match
• the percent meeting the states standard
• in each school
• to
• the distribution of NAEP plausible values
• in the schools NAEP sample.

14
e. g.,

• If 30 percent of the students in a school are
  reported by the state testing program to meet the
  states standard,
• Find the NAEP plausible value that has 30
  percent of the distribution higher than it.
• (If the NAEP mean in the school is 220, that
  value might be 230.)

15
in a school selected to participate in NAEP
o o o o o o o o o o o o o o o o o o o o o o o o o
o o o o o o o o o o o o o o o o o o o o o o o o o
o o o o o o o o o o o o o o o o o o o o o o o o o
o o o o o o o o o o o o o o o o o o o o o o o o o
o o o o o o o o o o o o o o
Grade 4 Students
o o o o o N o o o o o N o o o o o N o N o o o o o
o N o o o o N o o o o o N o N o o o o N o o o o o
o N o o o o o o o N o o o o o N o o o N o o o o o
o o o o o N o o o o N o o o o o N o o o o o o N o
o o o o o N o o o N o o o o
Some are selected for NAEP
o o o o o N o o o o o N o o o o o N o N o o o o
o o N o o o o No o o o o N o N o o o o N o o o
o o o N o o o o o o o N o o o o o N o o o N o
o o o o o o o o o N o o o o N o o o o o N o o
o o o o N o o o o o o N o o o N o o o o
Some meet state standard
16
in a school selected to participate in NAEP
o o o o o N o o o o o N o o o o o N o N o o o o
o o N o o o o No o o o o N o N o o o o N o o o
o o o N o o o o o o o N o o o o o N o o o N o
o o o o o o o o o N o o o o N o o o o o N o o
o o o o N o o o o o o N o o o N o o o o
o
N o o o N o
o o N o o o o o N o
o N o o o
N o o o o o o o N o o o
o o o o o N o o N o o o
o o o o o o o o N
o o o o o N o o o o o N o o o o o
o N o o o o o o N o o o o o o
N o o N o o o o o o o o N o o o o o o o o N o o
N o -----------------------------------------
Achievement Scale
o
N o o o N o
o o N o o o o o N o
o N o o o
N o o o o o o o N o o o
o o o o o N o o N o o o
o o o o o N o o o
o o o o o N o o o o o N o o o o o
o N o o o o o o N o o o o o o
N o o N o o o o o o o o N o o o o o o o o N o o
N o ------------------------------------------
Achievement Scale
Achievement frequency distribution in the school
17
in a school selected to participate in NAEP
30 of the distribution in one school
might correspond to 230 on the NAEP scale, in
that school.


n n n n
n n n n
n n n n n n n n n
n n n n n n n n
n n n n n n n n n
n n n n n n n n n n n n n n
n n n n n n n n n n n n n n n n
n n n n n n n n n n n n n n n n n n n n
n n n n n Achievement Scale
(NAEP Reading)
o
N o o o N o
o o N o o o o o N o
o N o o o
N o o o o o o o N o o o
o o o o o N o o N o o o
o o o o o o o o N
o o o o o N o o o o o N o o o o o
o N o o o o o o N o o o o o o
N o o N o o o o o o o o N o o o o o o o o N o o
N o Achievement Scale (State Reading
Assessment)
18
(No Transcript)
19
(No Transcript)
20
Assumption

• State reading assessment score variation
• is strongly related to
• NAEP score variation.

21
Assumption

• State reading assessment score variation
• is strongly related to
• NAEP score variation.
• If so, then we should get
• approximately the same estimate
• of the position of the state standard
• on the NAEP scale
• from nearly all NAEP schools in the state.

22
Test of the Assumption
23
Test of the Assumption
24
Test of the Assumption
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
Conclusions

• NAEP can provide a link between state achievement
  standards, especially at the proficient and
  advanced levels.
• There is substantial variation in standards
  between states.
• Percentages of students meeting state standards
  are lower in states in which the standards are
  set high.
• Most of the links for below proficient
  standards are weak.

36
Summary

• NAEP can contribute most to Leaving No Child
  Behind by helping state assessment programs give
  useful feedback to parents, teachers, and school
  administrators.