Interpreting PVAAS School Reports using a Growth Standard

1
PVAAS Growth Standard Methodology Statewide
Implementation
Pennsylvania Value-Added Assessment System (PVAAS)
Questions about these materials and the Growth
Standard Methodology can be directed to the PVAAS
Statewide Core Team. See contact info on Slide
31.
2
Important Questions for Value-added Calculations

• What is a Growth Standard and how is it set?
• How can we compare scores across different years?
• How do we estimate a students true level of
  achievment?

3
The Growth Standard Key Metric in PVAAS

• The Growth Standard specifies the minimal
  designated academic gain from grade to grade for
  a cohort of students.
• The use of a Growth Standard creates the
  possibility that ALL schools can demonstrate
  appropriate growth.

4
An Analogy
5
An Analogy

• Doctors plot a childs length/height over time.
• Each child may have a unique growth curve.

6
When growth acceptable?

• The length/height measurement is increasing over
  time.
• The length/height measurement maintains the
  approximate position in its length/height
  distributions as the child grows.
• The childs length/height continues to increase
  in a consistent manner.

A significant deviation of the growth pattern or
a change outside the typical range of values is
an indication that further investigation is
required.
7
What is the Growth Standard for a childs
length/height?

• The standard is that the child maintain the
  approximate same position each of the increasing
  distributions of length/heights as the child
  grows.
• A significant deviation from that pattern
  indicates a need for further investigation.

8
Growth Standard Charts for Academic Achievement

• Let us build an Academic Achievement Growth
  Chart.
• Collect the average performances of a large
  sample of students using a uniform assessment
  during each year of their career through school.
• Plot curves to represent appropriate percentile
  patterns.
• An example Suppose the following table
  represents the means and SDs of a group of
  students on the PSSA beginning in 3rd grade and
  continuing through 8th grade and ultimately 11th
  grade.

9
A Growth Standard Chart for Academic Achievement
10
An example of a cohorts growth
This cohorts mean performances have met the
Growth Standard since

• The growth curve approximately maintains its
  position in the distribution of scores.
• There are no significant deviations in the
  pattern of growth over time.

11
In an ideal world

• We would have a large body of longitudinal data
  from many cohorts to construct our growth charts.
• Since we do not, we will use the distributions
  from a base year for the creation of the growth
  curves.
• The base year distributions are approximates to
  the achievement distributions of a cohort from
  grade 3 to grade 8 and 11.

12
We use the base year distributions.
The base year for PVAAS is 2006.
13
Using the Base Year 2006

• Suppose the distributions from 2006 are given by

Conversion to NCE scores will use the Base Year
distributions in their calculations.
14
Suppose the means of a cohort in two consecutive
years are2007 3rd ? 1390 and 2008 4th ? 1450

• NCE scores are calculated for both using the 2006
  means and SDs.

2007 3rd ? 1390 ? 2008 4th ? 1450 ?
All future PSSA scaled scores will be converted
to NCE scores using the 2006 Base year parameters
for the comparison to calculate the mean gain of
a cohort of students.
15
The NCE Growth Curves
16
Some Thoughts

• This Growth Standard concept demonstrates the
  need for longitudinal data when considering
  academic growth since each student has his/her
  own academic growth curve.
• But
• The example also exhibits the remaining two
  issues for PVAAS value-added methods
• Comparing scores from year to year
• Estimate the true level of achievement for
  input into the growth curve.

17
Calculation of Gain from year to year

• Student growth is measured by difference in
  performance in consecutive years.

But there is a problem with this! These scores
are not comparable!
18
Comparing scaled scores on the PSSA from
different years

• PSSA tests have different means and standard
  deviations at each grade and for different years.
  For example, in 8th grade

19
A Solution Conversion to NCE Scores

• NCE scores indicate the position of a scaled
  score on a reference scale (mean 50, sd
  21.06) so that the scaled scores from different
  distributions with different scales can be
  compared.
• The use of NCE scores does not impose a normal
  distribution on the data, nor does the use of NCE
  scores have any relationship to normed referenced
  tests.
• NCEs are excellent for looking at scores over
  time.
• Using Data to Improve Student Learning in High
  Schools
• Victoria L. Bernhardt

20
NCE Scores Are About Position

• To calculate an NCE score
• Calculate the z-score of the data value of
  interest, that is, the number of standard
  deviations the data value is from the mean of its
  distribution
• The NCE score is calculated using the following
  formula

21
A Question

• George scores a 655 on the SAT mathematics exam.
• George also scores a 28 on the ACT mathematics
  exam.

• Which score should he report to his colleges if
  he wants to provide the better score?

22
A Matter of Comparison
The nature of each distribution is irrelevant to
the question of interest

• How do we compare Georges scores?

23
A Solution

• Conversion of both scores to NCE scores allows
  for the identification of the position of each
  score on the same scale.
• This identification of position provides the
  capability of comparison since the converted
  scores will be based on the same distribution
  parameters.

24
Which Score Should George Choose to Report?
Using a NCE scale with mean 50 and standard
deviation 21.06 SAT score of 655 ? NCE score
75.85 ACT score of 28 ? NCE score 80.74
ACT score
SAT score
Clearly, he should report his ACT score!
25
Consider Another Hypothetical Scenario
In 2006, Wilma was in 4th grade and scored as
follows on the 4th grade PSSA Mean for 4th Grade
2006 1303.24 Standard Deviation for 4th
Grade 2006 164.20 Wilmas scaled score
1425
In 2005, Wilma was in 3rd grade and scored as
follows on the 3rd grade PSSA Mean for 3rd Grade
2005 1356.75 Standard Deviation for 3rd
Grade 2005 126.20 Wilmas scaled score
1425
Do these scores indicated that Wilma progressed
during 4th grade?
26
Lets Look at it Graphically
Wilma
Wilma
Even though Wilmas scaled scores were the same
(both 1425), since the distributions were
different, we really cant compare the two scores
27
A Tentative Solution Conversion to Percentiles
Wilma
Wilma
In our example, Wilma score of 1425 was in the
66th percentile for 2005 but was in the 76th
percentile for 2006. These percentiles focus on
Wilmas position in each distribution.
28
But

• We cannot calculate Wilmas gain the difference
  of percentiles does not make sense
• Percentiles are not meaningful for calculating
  means for different years, gains, etc., since
  they are calculated from different distributions.

29
The Complete Solution Conversion to NCE Scores

• To establish a basis of comparison for different
  distributions from different schools in different
  years, we convert the scaled scores to units in
  the SAME scale.
• The scale we will use is from the NCE
  distribution with mean 50 and standard deviation
  approximately equal to 21.06.

Mean
30
The NCE Distribution and Wilma

• Wilmas NCE score for 2005 (3rd grade) is 61
• while her score for 2006 (4th grade) is 66.

Wilma 2006 4th
Wilma 2005 3rd
31
Wilmas gain

• Wilmas gain 2006 NCE score 2005 NCE score
• (4th Grade)
  (3rd Grade)
• 66
  61
• 5
• The mean gain of all of the students in Wilmas
  cohort can now be compared to the Growth Standard
  for growth for Wilmas cohort.

32
PVAAS Statewide Methodology
Student A Base Year NCE Score (2006)
2009 Observed School Mean NCE Scores
33
The Problem with the Mean of the Observed Scores

• The mean of the observed NCE scores at best
  represents a single snapshot in time of student
  achievement of the PSSA Anchors
• Is it the most comprehensive assessment of the
  schools TRUE level of achievement?
• How about the Bad Day syndrome?

34
Observed vs. Composite EstimateWhich is better?

• What if we combined the new, observed data with
  all of the prior PSSA assessment information that
  we have for this cohort of students?
• Would not a longitudinal view of the cohorts
  performance yield a more precise and reliable
  estimate of the true level of achievement?
• This is the essence and power of the
• PVAAS methodology!

35
Consider an Example

• Determine the percent of candies that are blue

If you were to open only one bag and find that
13 of the candies are blue, how much confidence
would you have in your estimate of the true
percentage of blue candies for all candies?
36
Only One Sample? A Bit Risky

• Lets open 50 bags and look at the distribution
  of the percents of blue candies

Looking at these 50 bags, what would you estimate
the true percent of blue candies for all
candies?
37
What If?
Distribution with n 50

• Lets open 50 more bags and add them to the 50
  selected earlier

Distribution with n 100
With this additional data, we can make a better
estimate of the true percent of blue candies!
38
The Function of Estimates

• The PVAAS methodology provides estimates of
  current and previous achievement, and subsequent
  gain for the school entity using all information
  for each student, no matter how complete or
  sparse.
• This process yields fair estimates of the impact
  of schooling on the rates of progress of the
  student populations and mitigates the problem of
  student mobility.

39
PVAAS Statewide Methodology
2009 Observed School Mean NCE Scores
2009 Estimated School Mean NCE Scores
Computer
2008 Estimated School Mean NCE Score
Gain 2009 Estimate 2008 Estimate
2007 Estimated School Mean NCE Score
2006 Estimated School Mean NCE Score
Compare to Growth Standard ? School Rating
40
How to Measure Growth of a School?

• Using a Growth Standard
• Student scaled scores are converted to NCE scores
  (2006 parameters).
• The mean NCE score for each school is calculated.
• PVAAS revises all earlier estimates based on the
  addition of the current data.
• PVAAS calculates an estimated NCE mean score.
• Estimated Mean NCE Gain
• Current Estimated NCE mean Previous Estimated
  NCE mean
• Gain is compared to Growth Standard for School
  Effect Rating.

41
Here is the Fall 2006 PVAAS District/School
Report
42
Gain Ratings
Mean NCE Gain for a cohort in a given year
represents the progress of students in that
cohort relative to the Growth Standard of
0. Color ratings Green mean gain greater than
or equal to the Growth Standard? favorable
indicator Yellow mean gain less than one SE
below the Growth Standard ? warning
sign Light Red mean gain is between one and
two SEs below the Growth Standard ? stronger
caution Red mean gain less two SEs below
the Growth Standard ? most serious warning
43
Level of Evidence The Role of Standard Error

• The color-coded ratings on the mean gain of
  cohorts are based on the level of confidence we
  have that the gain of the cohort is truly below
  the Growth Standard

44
The Power of PVAAS

• The power of this methodology is that it
  produces
• Accurate estimates of the true level of
  achievement of the students in this school.
• Updated estimates of all prior mean performance
  estimates simultaneously as new data is input
  into the longitudinal data structure.
• Over time, more accurate and reliable estimates
  of the true level of understanding of the
  students in this grade or school.

45
Questions?

• For more information, contact
• pdepvaas_at_iu13.org
• 717-606-1911

46
www.pde.state.pa.us
Gerald L. Zahorchak, D.Ed.
Secretary of Education
Commonwealth of Pennsylvania