Using PVAAS Data for School Improvement Planning

1
Using PVAAS Data for School Improvement Planning

• PVAAS Statewide Core Team
• August, 2008

PA SAS Standards-Aligned Systems
2

• Who is new to PVAAS?
• What do you want to find out about PVAAS today?
• Goal PVAAS will be used by teachers and
  administrators for annual school evaluation and
  planning, grade/subject level evaluation and
  planning, and for student level planning.

3
PDE Provides The Data Tools
4
Review of Literature

• While there is a relationship between
  achievement and demographic variables, there is
  NO relationship between growth and demographic
  factors, such as socioeconomic status and
  racial/ethnic background.
• --VAAS can remove the effects of factors not
  under the control of the school. (McCaffrey,
  Lockwood, Koretz Hamilton, 2003 Ross, Wang,
  Sanders, Wright Stringfield, 1999a Wright,
  Horn Sanders, 1997).

5
What Is PVAAS?
6
PVAAS IS

• A statistical analysis of existing assessment
  data.
• A realistic measurement of a schools
  progress/growth over time.
• Progress data to add to achievement data as
  schools are analyzing their work with students
  and making improvement plans.

7
PVAAS

• Is not another test.
• Is not being used for teacher-level
  accountability in Pennsylvania.
• Does not use teacher-level data elements.
• Does not yield a metric on the effectiveness of a
  teacher.

8
Current Data Collection Approaches
High Achievement
Average Achievement
Low Achievement
Achievement is based on each students ending
point.
9
PVAAS Adds Another Variable To AchievementProgres
s
Below Growth Standard
At or Above Growth Standard
Progress is based on each students starting
point.
10
The Scatter Plot Coordinate System
Vertical Axis PSSA Percent Proficient or Advanced
Growth Standard Boundary
AYP Percent Proficient Target
Horizontal Axis PVAAS Growth Measure
11
Scatter Plot Quadrants and Their Meanings
Quad 2 Adequate Achievement Negative Growth
Quad 1 Adequate Achievement Positive Growth
Quad 4 Below Adequate Achievement Positive Growth
Quad 3 Below Adequate Achievement Negative Growth
12
Example Regional Scatter Plot
13
Example Regional Scatter Plot Same Schools
14
The TWO PVAAS Methodologies
Looking Forward/Planning PVAAS Projection
Reports For Individual Students and Cohorts of
Students
Looking Back/Evaluation Value-added Growth
Reports For Cohorts of Students
Today
NO methodology to consider teacher effectiveness.
15
PDE Provides The Framework

• Getting Results!
• A Framework for Continuous School Improvement
  Planning
• The framework provides guiding questions for
  root cause analyses in the following data
  environments
• PSSA and AYP inquiry
• PVAAS
• 4Sight Benchmark Data
• Local Assessment Data Sets

16
Getting Results!

• Getting Results! is a TWO-year plan for
  continuous school improvement.
• The questions concerning data analysis that are
  posed in Getting Results!, Generation 5 will be
  addressed in this session.
• These are the questions schools answer in Year
  One of the two-year plan.

17
Year 1
Phase 1 Organize and Review the Data
Phase 2 Analyze Data and Discover Root Cause
Continuous Improvement Process
Phase 3 Plan Solution
Phase 7 Implement the Revision
Phase 4 Implement the Plan
Phase 6 Revise the Plan
Phase 5 Analyze Evidence of Effectiveness
Year 2
18

19
Annual School Level Planning

• For those schools not involved in the formal
  school improvement process, Getting Results!
  provides a good framework and guiding questions
  to assist your school in annual school level
  planning.
• The questions that we will address should still
  be the focus of the Analysis/Discovery phase of
  the Data-Informed Inquiry cycle that follows on
  the next slide.

20
Context 3-Phase Data-Informed Inquiry Cycle
Analysis Discovery
Data
Solutions
21
School Structures for Data-Informed Decision
Making
District-Level Support (Budgetary Support,
Professional Development, Resources and Time)
Student Learning Data
Demographic l Perceptual l Process Data
Annual Building-Wide Planning Process Focus All
Students Who School-Wide Team How PDE Getting
Results, Data Retreat, School/Continuous Planning
Process

• Building Level
• School Demographic Data
• PennData
• Discipline Data
• Attendance Data
• Mobility Rate Data
• Parent Surveys

• Building Level
• PSSA PVAAS
• Final 4Sight Benchmark Test
• Standardized Assessments
• District End-of-Year Tests
• EAP/Tutoring Assessments

• Grade/Course Level
• Initial PSSA/PVAAS/Final Tests
• Class/Subgroup Levels
• Cyclical
• 4Sight Benchmark Data Grade Level
• District Quarterly Assessments
• Common Classroom Data
• Classroom Summaries
• EAP/Tutoring Assessments

Periodic Grade-Level Planning Process Focus
Groups of Students Who Teacher Teams How
Regular 1-2 Hour meetings

• Grade/Course Level
• Class Demographic Data
• Class Engagement Data
• Satisfaction Data
• Attendance Data
• Walk Through Data

Student-Planning Process Focus Classroom of
Students Who Teacher

• Classroom Level
• Initial PSSA/PVAAS/Final Tests
• Student-Level Achievement and Growth Data
• Cyclical
• 4Sight Benchmark Data Student Level
• Continuous
• Individual Classroom Assessments
• EAP/Tutoring Assessments
• Progress Monitoring

• Classroom Level
• Qualitative Data
• Student Historical Information
• Student Medical Information
• Student Learning Information

22
Annual School ImprovementPlanning Meetings

• Unit of Analysis The School

23
Unit of Analysis The School

• Purpose of Analysis
• Plan to Improve Student Achievement and Meet AYP
  Targets
• Annual School-Wide Goal Setting
• Ensure instructional program coherence (IPC)
• Curriculum
• Instruction
• Assessment
• Supports (infrastructure including scheduling,
  staffing, etc)
• Monitoring Implementation and Effectiveness of
  School Plan

24
What Does Annual School-Level Planning Look Like
Now?

• Does it occur annually?
• Who is at the table?
• Principals
• Teachers
• Others
• What data are used?
• Does this result in an action plan?
• Is the action plan followed?

25
Data Packet Annual School-Level Analysis
26
1) Has the school met its AYP targets?
Data tool PA AYP Performance Chart
27
2) Has every grade met its AYP targets?
Data tool eMetric Organization Analysis of
Your PSSA Data - 3-year Portrait
Percent Scoring At/Above Proficient PSSA Reading
Results for Most Recent Year relative to NCLB/AYP
Target
28
3) Has every grade met or exceeded a years worth
of growth?
Data tool PVAAS School Value-Added Report
29
Value-added Measure ofCohort GrowthGrades 4
through 8 in Reading and Math

• Comparison to a State Growth Standard
• The Growth Standard specifies the minimal
  acceptable academic gain from grade to grade for
  a cohort of students.
• The use of a Growth Standard creates the
  possibility that ALL schools can demonstrate
  appropriate growth.

30
The Assessment Dilemma
True Student Achievement
Any test is just a snapshot in time!
31
Four Important Issues
Difference in scores in consecutive years.
1. Measuring Gain
2. Growth Standard
Convert all scores to same scale NCE scale
3. Comparing Scores from year to year
Combine current observed scores with all previous
scores in students histories.
4. Estimating Achievement
32
PVAAS Value-added CohortGrowth Descriptors for
Grades 4-8
33
Your Turn! Interpret this report!
34
PVAAS School ReportHigh School

• Based on a predictive methodology
• A predicted score is calculated for each student
• Mean of predicted scores is calculated for
  students in a cohort
• Mean of observed scores of students in the
  cohort
• School Effect is the difference between the
  Observed and the Predicted Scores.

35
PVAAS Value-added CohortGrowth Descriptors for
Grade 11
36
PVAAS School Value-Added Report,Grade 11
37
Check for UnderstandingValue-Added Report

• What do the four colors (green, yellow,
  rose/pink, and red) mean for grades 4-8? How is
  that different from the three colors (green,
  yellow, red) for grade 11?
• Why does the State Average NCE have a value of 50
  every year on the grades 4-8 reports?
• What does the Growth Standard mean for the grades
  4-8 reports? Why does it have a value of 0 every
  year?
• What is the mean predicted score on the grade 11
  report?
• Should a school only dig deeper if the gain is
  NOT green? Should a school only be concerned
  about gains that are not green?
• Is it appropriate to look at growth for the same
  grade level across the last several years
  (looking vertically at growth across years in a
  particular grade level)?

38
Hands-On with Value-added Reports

• Review by Grade Level
• Review by Subject
• Discuss Meaning as It Relates to AYP Targets
• Discuss as It Relates to Other Building/District
  Goals
• Discuss When These Would Be Used by Grade Level
  Teams

39
4) Is the trend over the past 3 years for each
grade an increase in percent Proficient or
Advanced?
Data tool eMetric Organization Analysis of
Your PSSA Data 3-year Portrait
Percent Scoring At/Above Proficient
40
5) Has every performance level in each grade met
or exceeded a years worth of growth?
Data tool PVAAS School Performance Diagnostic
Report
41
Performance Diagnostic Report
Every performance levels (below basic, basic,
proficient, advance) in each grade has met or
exceeded a years worth of growth? (Getting
Results!, p. 12)
42
PVAAS Performance Diagnostic ReportThe Key is
the RED Whisker!
43
What the whiskers tell us
Exceeded Growth Standard More than One Years
Growth Green
Met the Growth Standard Made One Years Growth
Yellow
Growth Standard (One Years Growth)
Below Growth Standard Less than One Years
Growth Rose
44
(No Transcript)
45
Greater Detail based on of students
46
Click on Hot Keys for lists of students.
47
Student Lists from Performance Diagnostic Reports
48
Check for UnderstandingPerformance Diagnostic
Report

• What do the Gain and Standard Error values mean?
• What do the colors mean on the pie chart version
  of this report?
• What does the red whisker, or red line, mean?
  How does it help in the interpretation of this
  report?
• What do the blue bars represent versus the gold
  bars?

49
Hands-On with Performance Diagnostic Reports

• Review by Grade Level
• Review by Subject
• Be Sure You Know How to Locate Subgroup Reports
  as well
• Discuss Patterns and Meaning of Patterns

50
6) Have subgroups met or exceeded the NCLB target
of Proficient or Advanced?
Suggested data tool eMetric Organization
Analysis of Your PSSA Data - 3-year Portrait
Percent Scoring At/Above Proficient PSSA Reading
Results for Most Recent Year relative to NCLB/AYP
Target
51
7) Has the achievement gap between the entire
student group and relevant subgroups become
narrower this year?
Data tool eMetric Organization Analysis of
Your PSSA Data - 3-year Portrait
Percent Scoring At/Above Proficient
52
8) Is each grade level on a trajectory to reach
the AYP goal at the end of this school year?
Data Tool PVAAS Grade Projection Summary Report
53
Student Projections

• Wouldnt it be great
• to know the likelihood
• that a student will
• be proficient on a
• future PSSA?

54
Student Projection Methodology

• A separate longitudinal modeling process focused
  on the performance scores of individual students.
• The model uses all of the available data (both
  math and reading) to project future PSSA
  performances in Math, Reading, Science and
  Writing.
• Projections can be calculated for an individual
  student or groups of students.

55
PVAAS Fall 2008 ReportingReading and Mathematics

• Projections to Basic, Proficient, and Advanced
• 3 ? 4 or 5
• 4 ? 5 or 6
• 5 ? 6 or 7
• 6 ? 7 or 8
• 7 ? 8
• 8 ? 11

Grade Last Tested
56
PVAAS Student Projection Report
57
Projection Summary Reports

• What is it?
• Report that summarizes the numbers and percents
  of students in likelihood ranges of performing at
  the proficient level on a future PSSA exam.
• How might a school use this report?
• Intervention Planning
• Resource Allocation
• Strategic Planning
• School Improvement Planning
• Cautions
• This report provides another indicator about
  likelihood of future performance.

58
Grade Projection Summary Report
59
Check for UnderstandingPVAAS Projections

• Can you get projections for just one year in to
  the future? Or, can you get projections for
  other years?
• How should a school interpret the three ranges of
  probabilities (i.e., 0-40, 40-70, 70-100)?
• How reliable are the projections?

60
Hands-On with Grade Projection Summaries

• Review by Grade Level
• Review by Subject
• Discuss Meaning as It Relates to AYP Targets
• Discuss as It Relates to Other Building/District
  Goals
• Discuss When These Would Be Used by Grade Level
  Teams

61
9) Has the school met its AYP participation
targets for all relevant subgroups?
Data tool PA AYP Data Table
62
10) Has every subgroup in each grade met or
exceeded a years worth of growth?
Data tool PVAAS School Performance Diagnostic
Report (by subgroup)
63
11) Have subgroups performed similarly toall the
student groups in each reporting category?
Data tool eMetric Group Summary Reporting
Categories
64
Summarize the patternsand trends.

• Summarize the patterns and trends in your data.
• What insights have you gained about the
  achievement and growth of students?

65
Activity 1

• Plot each grade, considering achievement
  (eMetric) and growth (PVAAS).
• Into what quadrant does each grade fall?
• Where are the strengths?
• Where are the areas for improvement?
• What questions does this raise about
• Curriculum
• Instruction
• Interventions
• Professional Development
• Resources

66
Activity 1
67
Activity 1
68
Activity 2

• Chart for each grade, the predicted performance
  levels with more than one years growth (star)
  and those with less than one years growth
  (circle)?
• Which groups are showing positive growth?
• Which groups are losing ground?
• Is this consistent across grade levels?
• What questions does this raise about
• Curriculum
• Instruction
• Interventions
• Professional Development
• Resources

69
Activity 2
Growth Standard (One Years Growth)
70
Activity 2
Growth Standard (One Years Growth)
71
Activity 3

• Chart for each grade and subgroup, the predicted
  performance levels with more than one years
  growth (star) and those with less than one years
  growth (circle)?
• Which groups are showing positive growth?
• Which groups are losing ground?
• Is this consistent across grade levels?
• How does the growth of the subgroups compare to
  overall students?

72
Activity 3
Growth Standard (One Years Growth)
73
Activity 3
Growth Standard (One Years Growth)
74
Discussion

• Is the school projected to meet AYP?
• What is the percent of students with high
  likelihood to reach proficiency?
• Is each grade projected to meet AYP?
• At each grade level, what is the percent of
  students with high likelihood to reach
  proficiency?
• How does this compare to the data you have just
  reviewed that looks back?
• Are students moving towards proficiency?

75
Discussion

• What insights have you gained about the
  achievement and growth of students?
• What questions are you left with concerning
• Curriculum
• Instruction
• Interventions
• Professional Development
• Resources

76
Next Steps 3-Phase Data-Informed Inquiry Cycle
Analysis Discovery
Data
Solutions
77
Year 1
Phase 1 Organize and Review the Data
Phase 2 Analyze Data and Discover Root Cause
Continuous Improvement Process
Phase 3 Plan Solution
Phase 7 Implement the Revision
Phase 4 Implement the Plan
Phase 6 Revise the Plan
Phase 5 Analyze Evidence of Effectiveness
Year 2
78
Data Packets Grade Level
79
Data Packets Student Level
80
Additional Professional Development

• Fall 2008 Professional Development
• 21 statewide webinars for districts
• Dates on PDE website
• 42 days of hands-on professional development
  offered at IUs
• Dates on PDE website
• Contact your IU for registration
• Follow-up support
• Professional Development Materials
• PDE website

81
Statewide PVAAS Professional Development, Support
Materials

• Contact Information
• pdepvaas_at_iu13.org
• 717-606-1911

PVAAS Website https//pvaas.sas.com
82
www.pde.state.pa.us
Gerald L. Zahorchak, D.Ed.
Secretary of Education
Commonwealth of Pennsylvania