Fidelity of Implementation in Scalingup Highly Rated Science Curriculum Units for Diverse Population

1
Fidelity of Implementation in Scaling-up Highly
Rated Science Curriculum Units for Diverse
Populations
Do not cite, quote or distribute without
permission from authors.

• Seminar

Prepared for National Science Foundation American
Geological Institute Instructional Materials
Development Conference

• Carol ODonnell
• Sharon Lynch, Ph.D.
• The George Washington University
• Washington, DC
• February 28, 2005

2
A little about my background

• I have the perspective of a teacher - educator
  for 22 years, K-12 classroom teacher for 10 years
  with current part-time faculty experience. I
  know its not always easy to teach with
  fidelity.
• I have the perspective of a curriculum developer
  - 11 years at NSRC developing 6 units for STC
  STC/MS. I believe in the materials Ive
  developed. Theyve been field tested. They can be
  effective if taught with fidelity.
• I have the perspective of a doctoral researcher -
  GWU, 2 years as Project Director Senior
  Research Associate of a large 5m implementation
  study. Dissertation focus will be FOI. How can
  we be assured the Treatment is being implemented
  as planned?

3
SCALE-uP Research Design

• 5-year 5.7M study funded by NSF/IERI with GWU
  and Montgomery County Public Schools (MCPS) as
  collaborators.
• Quasi-experimental design with matched pairs of
  middle schools (MCPS has 37 middle schools).
• Outcome measures include assessment of target
  benchmark/standard for each grade level (6th,
  7th, and 8th).
• Emphasis on disaggregated data (gender, SES,
  ethnicity, special education, and ESOL status).

Source Lynch, Kuipers, Pyke, Szesze, 2005
4
The George Washington University SCALE-uP Study

• Do highly rated inquiry-based science units
  improve student outcomes more than the standard
  curriculum? (Explored via experimental methods).
  Does disaggregating data reveal important
  differences hidden by overall mean scores? (Years
  0, 1, 2, 3)
• How do units function among a diverse group of
  students? (Explored via ethnographic methods).
  (Years 0, 1, 2, 3, 4)

Source Lynch, Kuipers, Pyke, Szesze, 2005
5
The George Washington University SCALE-uP Study

• Experience--Do students in schools in the first
  year of implementation have better outcomes than
  those in the second year? (Years 0, 1, 2, 3)
• Scale--Do students in schools at small scale (5
  schools) have better outcomes than students in
  schools at large scale (37 schools)? (Years 3, 5)
• Fidelity of implementation (FOI)--Do students
  whose teachers enact the curriculum with fidelity
  have higher outcomes than those whose teachers
  enact with less fidelity of implementation
  (measured by a classroom observation instrument
  currently under development)? (Year 4)

Source Lynch, Kuipers, Pyke, Szesze, 2005
6
FOI Background

• The bridge between a promising idea and its
  impact on students is implementation yet,
  rarely are programs implemented with fidelity
  (Berman and McLaughlin, 1976).
• Regulations and mandates such as NCLB now
  maintain that educational programs must be proven
  effective.
• Such mandates have created renewed interest in
  the importance of Fidelity of Implementation
  (FOI).

Source Lynch, ODonnell, Ruiz-Primo, Lee,
Songer, 2004.
7
Literature Review FOI Definitions

• Determination of how well a program is
  implemented in comparison with the original
  program design (Mihalic, 2002).
• Degree to which teachers and other program
  providers implement programs as intended by the
  program developers (Dusenbury, Brannigan, Falco,
  Hansen, 2003).

8
Literature Review FOI Criteria (Dane
Schneider, 1998)

• Adherence Strict adherence to methods or
  implementation that conforms to theoretical
  guidelines.
• Dose Completeness and amount of program
  delivered.
• Quality of Delivery The way by which a program
  is implemented.
• Participant Responsiveness The degree to which
  participants are engaged.
• Program Differentiation The degree to which
  elements which would distinguish one type of
  program from another are present or absent.

Adapted from Dane Schneider (1998) Dusenbury,
Brannigan, Falco, Hansen (2003) Additional
Source Mobrey, 2004
9
Developing our own criteria

• We knew to expectand would geta wide range of
  fidelity from the teachers. So we wanted to know
  how to gauge this range of fidelity. In
  developing our criteria, we asked the following
  questions
• Did the teacher teach the unit as it was
  designed?
• Was the unit taught for the expected duration?
• Did the teacher use highly rated instructional
  strategies to implement the unit?
• Did the students receive the curriculum as the
  unit developer intended it to be received?
• Did the teacher teach lesson components in the
  time and sequence expected by the developer?

10
This led us to the following FOI Criteria for
the SCALE-uP Study

• Adherence - Extent to which the instructional
  materials script (including its content and
  processes) is delivered as designed.
• Duration - Number of lessons length of time.
• Instructional Strategies The extent to which a
  teacher uses highly rated instructional
  strategies when implementing a lesson.
• Student Responsiveness Extent to which students
  are aware of and respond to the instructional
  strategies used to teach a unit.
• Lesson Flow Time and sequence in which lesson
  components are delivered.

Source Lynch, ODonnell, Ruiz-Primo, Lee,
Songer, 2004.
11
With these criteria in mind we did this

• Developed a classroom observation protocol to
  capture the following two criteria instructional
  strategies and student responsiveness.
• Categories and indicators of the protocol were
  based on the Project 2061 Instructional Analysis.
• Included indicators for both the teacher
  (instructional strategies) and student (student
  responsiveness).
• The measure of these criteria was based on
  frequencyHow often and during what time of the
  lesson was each indicator observed during one
  classroom visit?

12
Protocol Based on Project 2061 Instructional
Analysis

• Seven categories, each focused on a specific
  aspect of instructional support
• Providing a Sense of Lesson and Unit Purpose
• Taking Account of Student Ideas and
  Misconceptions
• Engaging Students with Relevant Phenomena
• Developing and Using Scientific Ideas
• Promoting Student Thinking about Phenomena,
  Experiences, and Knowledge
• Assessing Progress
• Enhancing the Science Learning Environment and
  Promoting Curiosity for all Students

Source Project 2061 Instructional Criteria
Available www.project2061.org/tools/textbook/mgs
ci/crit-used.htm
13
Methodology

• Pilot classroom observations included the
  following
• Year 1 (2002-2003) 15 classrooms 8th gr (2
  visits each)
• Year 2 (2003-2004) 30 classrooms 6-7th gr (1
  visit each)
• Plus
• 15 8th gr classrooms (1 visit before the
  treatment was implemented and 1 visit during
  implementation for each classroom)

14
General Thoughts

• Preliminary observations show that some aspects
  of classroom implementation are consistent with
  the units instructional intenthigh fidelity.
  Others were notlower fidelity.
• Therefore, during the SCALE-uP study, classroom
  observations also served as an accountability
  check to see the extent to which teachers were
  actually implementing the treatment.

15
Inherent Challenges

• Observing both the teachers and the students
  simultaneously proved to be difficult.
  Reliability of instrument was low.
• Instrument was revised to concentrate solely on
  the teacher (use of certain instructional
  strategies).
• A separate student questionnaire was developed to
  capture student responsiveness. To what extent
  are the students aware of the instructional
  strategies used to teach the intervention?
  (Instrument currently under development.)

16
Teachers Asked Questions About FOI
In the meantime

• Can we modify the unit to meet the needs of
  diverse student populations (SPED, ESOL, etc.)?
• What if we have requirements to meet state
  indicators (e.g. vocabulary) not covered by
  unit?
• Can we use instructional practices we typically
  use in the classroom (e.g. exit cards, warm
  ups)?
• How do we deal with student behavior issues?
• Can we add supplemental readings?

Source ODonnell, Lynch, Merchlinsky, 2004
17
Teachers' Questions about FOI Prompted a Set of
FOI Guidelines

• Fidelity is.
• Adhering to unit and lesson purpose, goals, and
  objectives.
• Adhering to unit pedagogical approaches.
• Following lesson sequence.
• Using the recommended equipment or materials.
• Making an adaptation to the lesson that does NOT
  change the nature or intent of the lesson.

Source ODonnell, Lynch, Merchlinsky, 2004
18

Teachers' Questions about FOI Prompted a Set of
FOI Guidelines

• Fidelity is not.
• Reducing or modifying unit goals and objectives.
• Reconfiguring the lesson so that other
  instructional practices gradually replace parts
  of the new unit.
• Reducing student expectations inherent to the
  unit.
• Varying grouping strategies outlined in the unit.
• Changing the units organizational patterns.
• Substituting other curriculum materials or
  lessons for those described by the unit.

Source ODonnell, Lynch, Merchlinsky, 2004
19
Teachers Questions Prompted the South Beach Diet
Analogy

• The treatment is specified (there is a book on
  it).
• The protocol is specified (a book lists all of
  foods in the amounts that can be eaten).
• The duration of each phase of the diet is
  specified.
• Thought An individual knows when she/he is off
  or on the diet. Although some variations are
  tolerated (e.g. some people must have their
  morning coffee, no matter what the diet says), it
  is quite clear that daily variations (such as
  super sized Big Macs with fries and Coke) so
  clearly compromise fidelity that the person
  cannot be considered on the diet. So, it is
  important to know what daily variations can be
  tolerated while maintaining the intent of the
  unit.

20
Teachers also asked, Why study FOI?

• To explore how effective interventions might be
  scaled up across many sites (i. e. if FOI is a
  moving target, generalizability of scale-up
  research may be imperiled).
• To gain confidence that the observed student
  outcomes can be attributed to the scaled
  intervention.
• To gauge the wide range of fidelity with which an
  intervention may be implemented.

Source Lynch, ODonnell, Ruiz-Primo, Lee,
Songer, 2004.
21
In addition, our research team asked

• How does a measure of FOI account for
  pre-existing good teaching practices vs. those
  prompted by the instructional material?
• Can we expect teachers and students to only
  exhibit instructional strategies inherent to the
  intervention?
• Why are certain aspects of instructional delivery
  (such as lesson closure) consistently absent,
  despite unit support?

Source ODonnell, Lynch, Merchlinsky, 2004
22
These questions prompted a new definition of FOI

• Fidelity of implementation is
• the extent to which a program (including its
  content and process) is implemented as designed
• how it is implemented (by the teacher)
• how it is received (by the students)
• how long it takes to implement (duration) and,
• what it looks like when it is implemented (lesson
  flow).
• This definition (and its associated criteria) may
  help us to gauge a wide range of fidelity.

23
Next Steps Years 3 and 4

• Year 3 (this year)
• Finalize instruments for measuring duration,
  student responsiveness, and lesson flow.
• Visit 20 treatment classrooms (7th grade).
• Quantify classroom observation scores.
• Year 4 (next year)
• Develop instrument for measuring adherence.
• Visit 40 treatment classrooms (6th 7th grade).
• Analyze correlation between FOI criteria and
  student outcomes.

24
Final Word Implications for Curriculum Developers

• The teacher in me says I am more likely to
  teach well-designed instructional materials with
  fidelity.
• The developer in me says Well-designed
  instructional materials allow for individual
  classroom differences and anticipatethrough
  field testhow teachers might implement the
  materials and how students may receive them.
• The researchers on our team say How do we
  identify the range of fidelity for different
  instructional materials whose efficacy may be
  unproven? What daily variations can be
  tolerated? And finally, can developers help us
  to define intent a priori?

25
Scaling up Curriculum for Achievement,
Learning, and Equity Project (SCALE-uP)The
George Washington University2134 G. St.,
NWWashington, DC 20052Telephone
202-994-4182Fax [phone]E-mail
codonnel_at_gwu.eduWeb www.gwu.edu/scale-up
This presentation was created by the