Washington State LASER Highlights From Recent Evaluation Studies

1
Washington State LASERHighlights From Recent
Evaluation Studies

• Dave WeaverRMC Research Corporation111 SW
  Columbia, Suite 1200Portland, Oregon 97201

2
Agenda For This Segment

• Impact on Student Achievement Studies
• Student-Level analysis of 2006 Science WASL
  results
• School-Level analysis of 2007 Science WASL
  results
• West Valley School District
• Implementation Study
• Study of Highly Successful Schools
• Evaluation activities for 2007-2008

3
Student-Level Analysis of 2006 Science WASL
Results
Spring Summer 2007
4
Student-Level Analysis of Grade 5 WASL Data for
2006

• Factors considered
• Student demographics (i.e., socioeconomics,
  race/ethnicity)
• Grade 5 science WASL results for 2006
• Grade 4 math and Reading WASL results for 2005
• The number of LASER PD hours the students Grade
  5 teacher received each year and prior to the
  administration of the 2006 WASL
• The number of LASER PD hours the students Grade
  4 teacher received each year and prior to the
  administration of the 2005 WASL.

5
2-Years of Teacher Influence
2006 Student-Level Science Data for Grade 5
PD Hours of Students Grade 5 Teacher in 2006
Linked By Teacher
Linked By Student
2005 Student-Level Math Data for Grade 4
PD Hours of Students Grade 4 Teacher in 2005
Linked By Teacher
6
Grade 5 Regression Analysis Results
Variable Beta t p
FRL 0.097 23.389 lt0.001
Hispanic 0.020 3.467 0.001
Asian 0.033 6.811 lt0.001
White 0.070 11.160 lt0.001
2005 Math Scale Score 0.688 174.346 lt0.001
06 05 Teacher PD 0.024 6.496 lt0.001
Dependent Variable 2006 Grade 5 Science Scale
Score
7
Factors Influencing 2006 Grade 5 Science WASL
Results
32,228 Students 860 Schools 1325 Classes R2 .645
8
Grade 8 Regression Analysis Results
Variable Beta t p
FRL 0.073 9.559 lt0.001
White 0.064 8.512 lt0.001
2005 Math Scale Score 0.747 100.039 lt0.001
06 05 Teacher PD 0.038 5.369 lt0.001
Dependent Variable 2006 Grade 8 Science Scale
Score
9
Factors Influencing 2006 Grade 8 Science WASL
Results
7,432 Students R2 .634
10
Finding 1

• After controlling for demographics and entry
  skill levels of students, the results reveal that
  the current year of LASER professional
  development is a significant positive predictor
  of scores on the science WASL results.
• If all things remain equal, 25 hours of
  professional development translates into a
  1-point gain in the science scale score.

11
Finding 2

• The professional development provided during the
  year that the teacher had the student in class
  had the greatest impact on student performance.
• The accumulated effect of previous years of
  professional development diminished predictive
  value.

12
Finding 3

• The current year of professional development of
  the 4th and 5th grade teachers is a significant
  positive predictor of student subscale scores
  for
• Inquiry in Science (.039)
• Open-ended responses (.029)
• Applications of Science (.020)

13
Finding 4

• Factors beyond the control of LASER (entry
  skills, demographics, poverty) account for 90 of
  knowable variance.
• Changes in any of these other factors can easily
  overshadow the positive effect of LASER
  professional development.

14
School-Level Analysis of 2007 Science WASL Results
Fall 2007
15
What We Did

• Data Sources
• School Level WASL Data from 2007
• LASER Professional Development Database
• School Demographic Data for 2006-07
• Regression Analysis
• Independent Variable
• Total PD hours of core teachers in a school for 2
  years prior to 2007 WASL per FTE
• Dependent Variable
• Percent of students in the school that met the
  Grade 5 science standard on the 2007 WASL
• Control Variables
• Percent of students who qualify for free or
  reduced price lunch
• Percent of students in the school that met the
  Grade 4 math standard on the 2006 WASL
• Students per classroom teacher

16
Grade 5 Regression Analysis Results
Variable Beta t p
2006 Grade 4 Math 0.481 11.294 lt0.001
FRL 0.400 9.425 lt0.001
Class size -0.088 -2.710 0.007
2-Year PD Per FTE 0.098 3.049 0.002
R2 .617
17
Findings

• The number of hours of LASER professional
  development per FTE over a 2 year period prior to
  the WASL is a significant predictor of student
  achievement on the Grade 5 science WASL above and
  beyond what can be explained by the number of
  students who qualify for free or reduced price
  lunch, class size, or student skills.
• This finding was not evident at Grade 8.

18
2-Year PD Hours Per FTE vs. Student Achievement
19
Quartile Grouping of Schools
PD Per FTE Rank Schools Adjusted Mean
3.75 or less 85 30.56
3.75 to 7.69 98 32.73
7.69 to 13.42 99 34.50
More than 13.42 103 37.01
20
Total School 2-Year PD Hours Per FTE Quartile
Ranking vs. Percent of Students Who Met The Grade
5 Science WASL in 2007
Note Means adjusted for Percent of students on
free or reduced price lunch, class size, and
previous year mathematics achievement.
21
Distribution of Study Schools By LASER Alliance
22
Analysis of West Valley School District Data
Fall 2005
23
West Valley School District study

• Peter Finch, Assistant Superintendent
• Pilot test involving phased implementation of
  inquiry-based instructional materials
• Noticed an important relationship between the
  number of modules students experienced and Grade
  5 science WASL scores.

24
Science WASL WVSD Grade 5 students meeting
standardStudents receiving LASER Science
instruction
State Average
25
Regression Analysis of 2004 Grade 5 Science WASL
Results
Variable Coefficient Significance
White 0.18 lt0.001
SPED -0.14 0.009
FRL N/A
Modules 0.15 0.003
26
Regression Analysis of 2005 Grade 5 Science WASL
Results
Variable Coefficient Significance
White 0.18 lt0.001
SPED -0.34 lt0.001
FRL -0.16 lt0.001
Modules 0.18 lt0.001
27
Conclusion

• The number of science modules that students
  experience has a significant positive impact on
  their science WASL achievement above and beyond
  what can be explained by demographic
  factors.(FRL, Special Education, White)

28
Study of Highly Successful Schools
Winter Spring 2007
29
Purpose of Study

• To identify schools that have demonstrated a
  consistent increases in the percentage of
  students who met the science standard on the
  Washington State Assessment of Student Learning
  (WASL).
• Interview the staff of those schools to determine
  to what they attribute their success.

30
Criteria for Selecting of Schools

• Focus on Grade 5
• Very few schools could demonstrate consistent
  improvement at Grade 8 and 10.
• At least 10 Grade 5 students complete the science
  WASL in 2004, 2005, and 2006.
• An increase of at least 6 or more students
  meeting the standard each year.
• An increase of at least 17.5 or more students
  meet the science standard across the 2 time
  intervals.
• Started with at least 20 of the students meeting
  the science standard in 2004.

31
Description of Selected Schools

• 40 schools selected for interview
• 18 (45) were in districts served by a LASER
  Alliance.
• 31 (78) were in districts that participated in
  at least 1 LASER Strategic Planning Institute
• 10 (25) are in districts that participate in one
  of the state MSP projects
• 4 (10) are participating in the NCOSP
• 1 school did not participate in the interviews

32
Interviews

• During December 2006 and January 2007.
• Conducted By
• RMC Research staff
• Graduate students from the science education
  department of Portland State University
• Using structure interview protocol
• Developed with input from LASER project
  leadership
• Training provided
• With person most knowledgeable about success
• Initial contact with Principal and sometimes
  other school staff

33
Two Tiered Approach

• Open ended initial question
• Summarized increases in WASL scores.
• What is your school doing to improve science
  teaching and learning? To what do you attribute
  this success?
• Followed by questions about the contribution of
  specific aspects such as
• Professional development
• Instructional materials
• Assessments
• LASER services

34
Finding 1 Instructional Materials

• All schools used inquiry-based instructional
  materials Mostly FOSS
• 49 (19) of the schools indicated that recent
  changes to the science instructional materials
  attributed to their success
• Some customized or locally developed kits
• Heavily supplemented
• Non-LASER Alliance schools use a wider variety of
  supplementary materials and approaches.

35
Finding 2 WASL Preparation

• Initially, 54 (21) of the schools indicated that
  they engage students in some form of preparation
  for the WASL.
• Upon questioning, 95 (37) indicated that they
  engage students in some form of preparation for
  the WASL.

36
Finding 2 WASL Preparation

• Most use WASL release items
• Prepare students before WASL administration
• Develop WASL-like practice assessment during the
  year
• Explicit emphasis on
• WASL format
• WASL vocabulary

37
Finding 3 Professional Development

• 82 (32) of the schools indicated that
  professional development contributed to improving
  student WASL scores
• PD in Non-LASER Alliance schools dominated by kit
  and WASL training
• PD in LASER Alliance schools is more varies
• Journaling/Notebooks
• Leadership
• Science Content
• Inquiry
• Coaching, Mentoring, Consultant
• EALRs and GLEs
• More Organized School-Based PD

38
Finding 4 Science Advocate

• 77 (30) of schools interviewed identified 1 or 2
  people who were advocates for science education.
• Passionate about science Made things happen
• Roles included
• Science specialist (16)
• Teacher leaders (9)
• Strong teacher or team (6)
• Community member (3)
• Administrator (2)

39
Finding 5 EALRs GLE

• 51 (20) of the schools attributed the success of
  their students to increased emphasis on and
  alignment of curriculum to the EALRs and GLEs.
• 11 are Non-LASER Alliance schools
• 9 are LASER Alliance schools

40
Finding 6 Instructional Time

• 38 (15) of the schools attributed their success
  at least partially to increased science
  instructional time
• 8 are Non-LASER Alliance schools
• 7 are LASER Alliance schools

41
Recommendations From Schools

• Interviewees were asked
• What 2 recommendations would you give to the
  staff of another school working to improve
  science teaching and learning that would improve
  student performance on the science WASL?
• Responses varied considerably

42
Top 6 Recommendations

• Use released items PCAs to develop skills
  needed to be successful on the WASL (18).
• Provide adequate professional development (13) on
  materials and WASL.
• Increase science instruction time (10).
• Engage students in hands-on, inquiry-based
  science activities (9).
• Provide teacher collaboration time (7).
• Align with EALRs and GLEs (6).

43
Conclusions

• LASER actively promotes most of the key findings
  from this study
• Inquiry-based instructional materials
• Preparing students for the science WASL
• Professional development
• Science advocacy (leadership)
• Alignment of curriculum to EALRs GLEs
• Increased science instructional time

44
Evaluation Plans for 2007-08
45
Sentinel Site Visits

• Identify of approximately 30 schools in 200708
  and 30 in 200809
• Schools With Significant LASER Participation
• Two Groups of 15 schools each
• Demonstrated significant positive gains
• Demonstrated little, no, or negative gains
• Each site visit 1 ½ to 2 days

46
Site Visit Data Collection

• Interviews with principals and/or science teacher
  leader
• Interviews with teachers
• Observations of at least 3 lessons where
  inquirybased instructional materials are used
• Based on STAR Protocol
• Rubrics for rating level of implementation of
  inquiry-based science
• Analyze results for significant relationships
  between level of implementation to student
  achievement

47
Assessment of Student Content Knowledge

• Obtain permission to use Bellevue School District
  assessments for Grades 6 7.
• Identify teacher participants
• 25 first time teachers 25 veteran teachers
• Data collection for each student
• Pre and post student scores
• Student demographics
• Special program participation
• SPED, FRL, ELL, Migrant, etc.
• Gift certificate compensation to teachers

48
Questions !

• Dave Weaver
• RMC Research Corporation111 SW Columbia, Suite
  1200Portland, Oregon 97201
• [phone]
• dweaver_at_rmccorp.com