Title: Fidelity of Implementation in Scalingup Highly Rated Science Curriculum Units for Diverse Population
1Fidelity of Implementation in Scaling-up Highly
Rated Science Curriculum Units for Diverse
Populations
Do not cite, quote or distribute without
permission from authors.
Prepared for National Science Foundation American
Geological Institute Instructional Materials
Development Conference
- Carol ODonnell
- Sharon Lynch, Ph.D.
- The George Washington University
- Washington, DC
- February 28, 2005
2A little about my background
- I have the perspective of a teacher - educator
for 22 years, K-12 classroom teacher for 10 years
with current part-time faculty experience. I
know its not always easy to teach with
fidelity. - I have the perspective of a curriculum developer
- 11 years at NSRC developing 6 units for STC
STC/MS. I believe in the materials Ive
developed. Theyve been field tested. They can be
effective if taught with fidelity. - I have the perspective of a doctoral researcher -
GWU, 2 years as Project Director Senior
Research Associate of a large 5m implementation
study. Dissertation focus will be FOI. How can
we be assured the Treatment is being implemented
as planned?
3SCALE-uP Research Design
- 5-year 5.7M study funded by NSF/IERI with GWU
and Montgomery County Public Schools (MCPS) as
collaborators. - Quasi-experimental design with matched pairs of
middle schools (MCPS has 37 middle schools). - Outcome measures include assessment of target
benchmark/standard for each grade level (6th,
7th, and 8th). - Emphasis on disaggregated data (gender, SES,
ethnicity, special education, and ESOL status).
Source Lynch, Kuipers, Pyke, Szesze, 2005
4The George Washington University SCALE-uP Study
- Do highly rated inquiry-based science units
improve student outcomes more than the standard
curriculum? (Explored via experimental methods).
Does disaggregating data reveal important
differences hidden by overall mean scores? (Years
0, 1, 2, 3) - How do units function among a diverse group of
students? (Explored via ethnographic methods).
(Years 0, 1, 2, 3, 4)
Source Lynch, Kuipers, Pyke, Szesze, 2005
5The George Washington University SCALE-uP Study
- Experience--Do students in schools in the first
year of implementation have better outcomes than
those in the second year? (Years 0, 1, 2, 3) - Scale--Do students in schools at small scale (5
schools) have better outcomes than students in
schools at large scale (37 schools)? (Years 3, 5) - Fidelity of implementation (FOI)--Do students
whose teachers enact the curriculum with fidelity
have higher outcomes than those whose teachers
enact with less fidelity of implementation
(measured by a classroom observation instrument
currently under development)? (Year 4)
Source Lynch, Kuipers, Pyke, Szesze, 2005
6FOI Background
- The bridge between a promising idea and its
impact on students is implementation yet,
rarely are programs implemented with fidelity
(Berman and McLaughlin, 1976). - Regulations and mandates such as NCLB now
maintain that educational programs must be proven
effective. - Such mandates have created renewed interest in
the importance of Fidelity of Implementation
(FOI).
Source Lynch, ODonnell, Ruiz-Primo, Lee,
Songer, 2004.
7Literature Review FOI Definitions
- Determination of how well a program is
implemented in comparison with the original
program design (Mihalic, 2002). - Degree to which teachers and other program
providers implement programs as intended by the
program developers (Dusenbury, Brannigan, Falco,
Hansen, 2003).
8Literature Review FOI Criteria (Dane
Schneider, 1998)
- Adherence Strict adherence to methods or
implementation that conforms to theoretical
guidelines. - Dose Completeness and amount of program
delivered. - Quality of Delivery The way by which a program
is implemented. - Participant Responsiveness The degree to which
participants are engaged. - Program Differentiation The degree to which
elements which would distinguish one type of
program from another are present or absent.
Adapted from Dane Schneider (1998) Dusenbury,
Brannigan, Falco, Hansen (2003) Additional
Source Mobrey, 2004
9Developing our own criteria
- We knew to expectand would geta wide range of
fidelity from the teachers. So we wanted to know
how to gauge this range of fidelity. In
developing our criteria, we asked the following
questions - Did the teacher teach the unit as it was
designed? - Was the unit taught for the expected duration?
- Did the teacher use highly rated instructional
strategies to implement the unit? - Did the students receive the curriculum as the
unit developer intended it to be received? - Did the teacher teach lesson components in the
time and sequence expected by the developer?
10This led us to the following FOI Criteria for
the SCALE-uP Study
- Adherence - Extent to which the instructional
materials script (including its content and
processes) is delivered as designed. - Duration - Number of lessons length of time.
- Instructional Strategies The extent to which a
teacher uses highly rated instructional
strategies when implementing a lesson. - Student Responsiveness Extent to which students
are aware of and respond to the instructional
strategies used to teach a unit. - Lesson Flow Time and sequence in which lesson
components are delivered.
Source Lynch, ODonnell, Ruiz-Primo, Lee,
Songer, 2004.
11With these criteria in mind we did this
- Developed a classroom observation protocol to
capture the following two criteria instructional
strategies and student responsiveness. - Categories and indicators of the protocol were
based on the Project 2061 Instructional Analysis. - Included indicators for both the teacher
(instructional strategies) and student (student
responsiveness). - The measure of these criteria was based on
frequencyHow often and during what time of the
lesson was each indicator observed during one
classroom visit?
12Protocol Based on Project 2061 Instructional
Analysis
- Seven categories, each focused on a specific
aspect of instructional support - Providing a Sense of Lesson and Unit Purpose
- Taking Account of Student Ideas and
Misconceptions - Engaging Students with Relevant Phenomena
- Developing and Using Scientific Ideas
- Promoting Student Thinking about Phenomena,
Experiences, and Knowledge - Assessing Progress
- Enhancing the Science Learning Environment and
Promoting Curiosity for all Students
Source Project 2061 Instructional Criteria
Available www.project2061.org/tools/textbook/mgs
ci/crit-used.htm
13Methodology
- Pilot classroom observations included the
following - Year 1 (2002-2003) 15 classrooms 8th gr (2
visits each) - Year 2 (2003-2004) 30 classrooms 6-7th gr (1
visit each) - Plus
- 15 8th gr classrooms (1 visit before the
treatment was implemented and 1 visit during
implementation for each classroom)
14General Thoughts
- Preliminary observations show that some aspects
of classroom implementation are consistent with
the units instructional intenthigh fidelity.
Others were notlower fidelity. - Therefore, during the SCALE-uP study, classroom
observations also served as an accountability
check to see the extent to which teachers were
actually implementing the treatment.
15Inherent Challenges
- Observing both the teachers and the students
simultaneously proved to be difficult.
Reliability of instrument was low. - Instrument was revised to concentrate solely on
the teacher (use of certain instructional
strategies). - A separate student questionnaire was developed to
capture student responsiveness. To what extent
are the students aware of the instructional
strategies used to teach the intervention?
(Instrument currently under development.)
16Teachers Asked Questions About FOI
In the meantime
- Can we modify the unit to meet the needs of
diverse student populations (SPED, ESOL, etc.)? - What if we have requirements to meet state
indicators (e.g. vocabulary) not covered by
unit? - Can we use instructional practices we typically
use in the classroom (e.g. exit cards, warm
ups)? - How do we deal with student behavior issues?
- Can we add supplemental readings?
Source ODonnell, Lynch, Merchlinsky, 2004
17Teachers' Questions about FOI Prompted a Set of
FOI Guidelines
- Fidelity is.
- Adhering to unit and lesson purpose, goals, and
objectives. - Adhering to unit pedagogical approaches.
- Following lesson sequence.
- Using the recommended equipment or materials.
- Making an adaptation to the lesson that does NOT
change the nature or intent of the lesson.
Source ODonnell, Lynch, Merchlinsky, 2004
18 Teachers' Questions about FOI Prompted a Set of
FOI Guidelines
- Fidelity is not.
- Reducing or modifying unit goals and objectives.
- Reconfiguring the lesson so that other
instructional practices gradually replace parts
of the new unit. - Reducing student expectations inherent to the
unit. - Varying grouping strategies outlined in the unit.
- Changing the units organizational patterns.
- Substituting other curriculum materials or
lessons for those described by the unit.
Source ODonnell, Lynch, Merchlinsky, 2004
19Teachers Questions Prompted the South Beach Diet
Analogy
- The treatment is specified (there is a book on
it). - The protocol is specified (a book lists all of
foods in the amounts that can be eaten). - The duration of each phase of the diet is
specified. - Thought An individual knows when she/he is off
or on the diet. Although some variations are
tolerated (e.g. some people must have their
morning coffee, no matter what the diet says), it
is quite clear that daily variations (such as
super sized Big Macs with fries and Coke) so
clearly compromise fidelity that the person
cannot be considered on the diet. So, it is
important to know what daily variations can be
tolerated while maintaining the intent of the
unit.
20Teachers also asked, Why study FOI?
- To explore how effective interventions might be
scaled up across many sites (i. e. if FOI is a
moving target, generalizability of scale-up
research may be imperiled). - To gain confidence that the observed student
outcomes can be attributed to the scaled
intervention. - To gauge the wide range of fidelity with which an
intervention may be implemented.
Source Lynch, ODonnell, Ruiz-Primo, Lee,
Songer, 2004.
21In addition, our research team asked
- How does a measure of FOI account for
pre-existing good teaching practices vs. those
prompted by the instructional material? - Can we expect teachers and students to only
exhibit instructional strategies inherent to the
intervention? - Why are certain aspects of instructional delivery
(such as lesson closure) consistently absent,
despite unit support?
Source ODonnell, Lynch, Merchlinsky, 2004
22These questions prompted a new definition of FOI
- Fidelity of implementation is
- the extent to which a program (including its
content and process) is implemented as designed - how it is implemented (by the teacher)
- how it is received (by the students)
- how long it takes to implement (duration) and,
- what it looks like when it is implemented (lesson
flow). - This definition (and its associated criteria) may
help us to gauge a wide range of fidelity.
23Next Steps Years 3 and 4
- Year 3 (this year)
- Finalize instruments for measuring duration,
student responsiveness, and lesson flow. - Visit 20 treatment classrooms (7th grade).
- Quantify classroom observation scores.
- Year 4 (next year)
- Develop instrument for measuring adherence.
- Visit 40 treatment classrooms (6th 7th grade).
- Analyze correlation between FOI criteria and
student outcomes.
24Final Word Implications for Curriculum Developers
- The teacher in me says I am more likely to
teach well-designed instructional materials with
fidelity. - The developer in me says Well-designed
instructional materials allow for individual
classroom differences and anticipatethrough
field testhow teachers might implement the
materials and how students may receive them. - The researchers on our team say How do we
identify the range of fidelity for different
instructional materials whose efficacy may be
unproven? What daily variations can be
tolerated? And finally, can developers help us
to define intent a priori?
25Scaling up Curriculum for Achievement,
Learning, and Equity Project (SCALE-uP)The
George Washington University2134 G. St.,
NWWashington, DC 20052Telephone
202-994-4182Fax 202-994-0692E-mail
codonnel_at_gwu.eduWeb www.gwu.edu/scale-up
This presentation was created by the