Studying Fidelity of Implementation (FOI): How FOI influenced SCALE-uP

1
Studying Fidelity of Implementation (FOI) How
FOI influenced SCALE-uPs Theory of Action for
Middle School Science Curriculum Materials

• SCALE-uP Scaling up Curriculum for
  Achievement Learning and Equity Project, a
  partnership between George Washington University
• and Montgomery County Public Schools
• Sharon Lynch, PI
• Co-PIs Curtis Pyke, Joel Kuipers, Michael
  Szesze Bonnie Hansen-Grafton
• http//www.gwu.edu/scale-up/
• Prepared for Researchers Without Borders Webinar,
  May 26, 2010

2
FOI Research Group

• Carol ODonnell
• Suzanne Merchlinski MCPS evaluation staff
• Bonnie Hansen-Grafton
• Joelle Lastica
• Vasuki Rethinam
• Bill Watson
• Rob Ochsendorf
• Liz Hatchuel
• Annie Hansen
• With special thanks to MCPS middle school science
  teachers who participated in this study

3
Disclaimer

• Indebted to the Interagency Educational Research
  Initiative (IERI) administered by the NSF, for
  research funding for SCALE-uP (7 year research
  program).
• I am currently working at NSF as a Program
  Director in EHR/DRL ( return to my position as a
  professor at GWU in September).
• However, the ideas and opinions discussed here
  are entirely my own and in no way represent the
  those of NSF.

4
SCALE-uP/FOI WebinarCautionary Tale w/ Happy
Ending

• Background of SCALE-uP and initial Theory of
  Action
• Year 0 Pilot Study Curriculum modifications are
  tricky business!
• Year 1 Comparison groups are (incredibly) handy
  in developing FOI instruments and understanding
  the study context
• Year 2 3 Comparison group and FOI evidence are
  crucial for credible evidence of effectiveness
• Year 4 Putting it all together How the Theory
  of Action was changed by FOI evidence FOI as
  process and structure constructs for both
  teacher and student

5
Background for SCALE-uP and Initial Theory of
Action

• In 1990s, AAAS Project 2061 developed a
  Curriculum Analysis to identify curriculum
  materials likely to help students learn a target
  idea (benchmark/standard).
• Curriculum Analysis relied on experts judgment
  of written curriculum materials.
• Two parts
• --Focused, accurate, coherent content on a
  standard/benchmark
• --Instructional strategies contained in
  written curriculum materials

6
Project 2061 Instructional Strategies for
curriculum materials

1. Convey sense of purpose
2. Address student ideas and misconceptions
3. Promote engagement with relevant phenomena
4. Developing, using scientific ideas
5. Encourage student thinking
6. Encourage assessment of progress
7. Creating positive learning environment
   curiosity, all students

AAAS. Project 2061.
7
Background for SCALE-uP

• Project 2061 Curriculum Analysis had located only
  2 acceptable curriculum units in middle school
  science.
• Units had been field-tested with small numbers of
  students (no comparison groups).
• Note. More mathematics curriculum materials had
  acceptable ratings and were field-tested and
  studied and scaled.

8
Background for SCALE-uP and Research Questions

• If science curriculum materials having Project
  2061 attributes were studied in a series of large
  (N 2000) quasi-experiments using carefully
  matched comparison groups
• Would they be effective?
• Would they be equitable?
• Would there be a relationship between fidelity of
  implementation to a unit and student outcomes?
• Could the materials be scaled-up in this large
  school district?
• How did they function in classroom
  (video-ethnography)?

9
SCALE-uPs Interventions 3 Science Units with
Coherent Focused on Target Ideas

• State of Michigans Chemistry That Applies (CTA)
  focuses on conservation of matter. 8th graders,
  unit 6 weeks long.
• GEMS Lawrence Hall of Science Real Reasons for
  the Season (Seasons) focuses on the reasons for
  the Earths seasons. 7th graders, unit, 3
  weeks.
• ARIES Harvard Smithsonian Motion and Forces (MF)
  focuses on portions of Newtons Laws. 6th
  graders, unit 6 weeks long.

10
Curriculum Analysis Instructional Strategies
? Excellent, ?Very Good, ?Satisfactory, ?Fair
?Poor
Instructional Category Chemistry That Applies ARIES Motion Forces GEMS Seasons Macmillan/ McGraw Hill
I. Identifying a Sense of Purpose
Conveying Unit Purpose ? ? NR ?
Conveying lesson/activity purpose ? ? ? ?
Justifying lesson/activity sequence ? ? ? ?
II. Taking Account of Student Ideas
Attending to prerequisite knowledge and skills ? ? ? ?
Alerting teacher to commonly held ideas ? ? NR ?
Assisting teacher in identifying own students ideas ? ? ? ?
Addressing commonly held ideas ? ? ? ?
11
? Excellent, ?Very Good, ?Satisfactory, ?Fair
?Poor
Instructional Category Chemistry That Applies ARIES Motion Forces GEMS Seasons Macmillan/ McGraw Hill
III. Engaging Students with Relevant Phenomena
Providing a variety of phenomena ? ? ? ?
Providing vivid experiences ? ? ? ?
IV. Developing and Using Scientific Ideas
Introducing terms meaningfully ? ? ? ?
Representing ideas effectively ? ? ? ?
Demonstrating use of knowledge ? ? ? ?
Providing practice ? ? ? ?
V. Promoting Student Thinking about Phenomena, Experiences, and Knowledge
Encouraging students to explain their ideas ? ? ? ?
Guiding student interpretation and reasoning ? ? ? ?
Encouraging students to think about what theyve learned ? ? ? ?
12
SCALE-uPs Outcome Measures

• Curriculum-independent measure for each unit
  focusing on the units target idea.
• Assessments had good psychometric properties and
  were developed using a Project 2061 assessment
  system.
• Multiple choice and constructed response items
  designed to be maximally accessible to students
  of varied language skills.

13
Questions?
14
Background for SCALE-uP and Initial Theory of
Action, c. 2001

• Curriculum units highly rated on Curriculum
  Analysis could be effective overall because
• each focused coherently on one big idea/standard/
  benchmark
• each had a carefully planned sequence of
  activities, and
• each contained identified instructional
  strategies leading students to construct
  understanding of one target idea/benchmark/standar
  d.
• Big Question Would they be equitable?
• Assumption Business as usual comparison
  classrooms would be less focused, rely more on
  textbooks and worksheets, and provide less time
  for guided inquiry and lab work.

15
Typical Theory of Action
Teacher
Curriculum Materials
Student Outcomes
16
Fidelity of implementationMowbray, Holter,
Teague Bybee, 2003

• Fidelity of implementation is the extent to
  which the delivery of an intervention adheres to
  the original program theory behind its
  development it confirms that the implementation
  of the independent variable in outcome research
  occurred as planned
• (and involves) the dynamic nature of fidelity
  criteria, appropriate validation and statistical
  analysis methods, the inclusion of structure and
  process criteria in fidelity assessment and the
  role of program theory in deciding on the balance
  between adaptation versus exact replication of
  model programs.

17
SCALE-uP Theory of Action c. 2001
Teacher FOI Instructional Strategies
Teacher
Curriculum Materials
Student Outcomes
18
SCALE-uP Theory of Actionc. 2001

• Teachers would need to locate and enact the
  instructional strategies embedded in the
  curriculum unit (identified via the Project 2061
  Curriculum Analysis).
• Doing this well would be to implement with
  fidelity.
• Hypothesis The better the implementation of the
  units embedded instructional strategies in a
  classroom, the higher the student outcomes.
• All SCALE-uP needed to do to create a classroom
  observation instrument that captured teachers
  implementation of the strategies.
• This would become Instructional Strategies
  Classroom Observation Protocol (more on this
  later).
• Simple!

19
Questions?

• On to Fidelity of Implementation (FOI)!

20
Year 0 (Pilot Study) Results Chemistry That
Applies (CTA) with Modifications Encouraged

• CTAs results showed effect sizes .52.
• Results disaggregated for subgroups of students
  showed that no students disadvantaged by CTA.
• Teachers were asked to modify CTA for diverse
  learners and record modifications.
• Virtually none did! This raised questions about
  whether to modify CTA in future. Teacher beliefs
  about the unit were mixed.
• Co-PI Szesze wanted to be sure units were
  unambiguously effective, or not.
• Decision to implement with fidelity in the
  future studies .
• Teachers and researchers drew up fidelity
  guidelines together.

21
Lesson Learned Modifying Curriculum Unit is
Tricky Business!

• If an intervention does not have solid evidence
  of efficacy/effectiveness, then modifications
  muddy the water CTA and the other units were
  unproven interventions.
• Therefore, their critical components could not be
  known, but assumed.
• In retrospect, this was a good decision for
  SCALE-uP capturing modifications while trying to
  establish if a unit worked would not be
  credible in this school district context.

22
Iterative Process of Identifying Critical
Attributes, Measuring Attributes, and Looking for
Relationships between FOI and Outcomes
Identify Critical Attributes of Intervention
FOI measure
Reconsider Critical Attributes
Outcomes
Mowbray, C., Holter, M. C., Teague, G. B.,
Bybee, D. (2003). Fidelity criteria Development,
measurement, and validation. American Journal of
Evaluation, 24(3),315-340.
23
Questions?

• On to Year 1
• Replication of CTA
• Building the first FOI Instrument

24
Yr. 1 Replication of CTA and Develop ISCOP/FOI
Process

• Given Theory of Action (the more teachers
  implemented Project 2061 instructional strategies
  embedded in curriculum units, the higher student
  outcomes), we began to develop an instrument that
  could capture fidelity to identified Project 2061
  instructional strategies.
• A generic instrument for all 3 units studied,
  but the units had been carefully vetted and had
  much in common.
• ISCOP (Instructional Strategies Classroom
  Observation Instrument), the first FOI measure
  was born, and developed over the next 4 years.

25
(No Transcript)
26
Effect Sizes CTA (Year 1)
27
Years 1, 2, 3, 4 Test ISCOP in Treatment and
Comparison Classrooms

• Was the ISCOP capturing fidelity to instructional
  strategies in a way that discriminated between
  Treatment and Comparison classrooms?
• ISCOP did not discriminate very well, so it was
  refined and refined and refined.
• Was ISCOP a bad instrument? Or were Treatment
  and Comparison classrooms similar for
  Instructional Strategies?
• If Treatment and Comparison classrooms were
  similar in strategies, should the Theory of
  Action be revisited?

28
Lesson Learned Comparison Classrooms Invaluable
as Counterfactuals

• ISCOP data suggested that more complexity than
  assumed in the Theory of Action.
• Measuring FOI relying solely with ISCOP might not
  answer FOI research question if comparison
  classroom data were taken into consideration.
• Generic measures of instructional strategies/FOI
  process are notoriously hard to developvalid and
  reliable?

29
Questions?

• On to Years 2 and 3 with a new unit, Real Reasons
  for the Seasons
• (Seasons)

30
Year 2 and 3 Results Comparison group outscores
Seasons group How to account for this?

• Seasons studied in 7th grade classrooms (N 2000
  students and 40 classrooms).
• Year 2 and 3 results Comparison group has higher
  outcome than Seasons on curriculum independent
  measure (ES -.36, -.18, respectively).

31
Comparison Classrooms Invaluable as
Counterfactuals

• Comparison classroom teachers surveyed and
  interviewed.
• Data showed that Comparison classrooms were
• -Non-traditional
• -Variety of curriculum materials used,
  including other inquiry units
• -Focused on the target idea
• -Equal Duration of Treatment and Comparison
  units
• -Comparison and Treatment teacher
  characteristics similar.

32
Seasons Replication in Year 3 Focus on FOI

• Use ISCOP in Treatment and Comparison classrooms.
• Develop a new Lesson Flow FOI process measure
  that gauged Teacher, Student-Group or Individual
  Centeredness of classrooms based on hunch that
  Student-Group Centeredness was important to
  student construction of science ideas in groups.
• Videotaped a Seasons and Comparison classroom.
• Interviewed and surveyed Seasons and Comparison
  Teachers again.

33
Lesson Flow Classroom Observation Instrument
34
Effect Sizes Seasons (Year 3)Overall ES -.18
35
Year 3 FOI Results

• Only 3 items on ISCOP showed significant
  differences between Seasons and Comparison
  classrooms, and 2 of 3 were observed more often
  in Comparison classrooms.
• Lesson Flow Seasons classrooms were more
  teacher-centered than Comparison
• Teacher-Centeredness
• Seasons 71 of time
• Comparison 58 of time
• Video-data backed this up.
• Emerging Conjecture Students need time in to
  work and talk in groups to develop their ideas
  and Seasons allowed less time for this.

36
SCALE-uP Theory of Action c. 2006
Teacher FOI Instructional Strategies
Teacher
Curriculum Materials
Student Outcomes
Student FOI Student Group Centeredness
37
Changing Theory of Action and Approach to FOI c.
2006

• Focusing FOI entirely on teacher in a guided
  inquiry unit may be simplistic.
• Student agency implicated in FOI constructs.
• But Lesson Flow is controlled by the teacher who
  provides students with the time/space to work in
  groups students can choose to do the science
  work, or not.
• Lesson Flow is not Time On Task.

38
Seasons Unit and FOIThe Research Re-visited

• Seasons was designed to be a supplementary unit.
• Developers seemed not to understand the
  implications of this when agreement was made to
  study effectiveness/FOI.
• SCALE-uP researchers did not appreciate the
  differences in philosophy of Seasons with
  research design.
• Seasons was a bad match for this study.
• Revisions made to Seasons based upon feedback
  from MCPS teachers.

39
Questions?

• On to Year 4 and Motion and Forces
• (MF)
• Note this is the third curriculum unit studied,
• not to be confused with the first two, and was
  implemented in 6th grade classrooms.

40
MF Results in Years 2 and 3Underwhelming

• Year 2 ES .10
• Year 3 ES -.06
• FOI had not been emphasized, so there was little
  information to explain lackluster outcomes.
• However, we learned that students had not been
  issued MF student Journals in Years 2 and 3.
  Was this an important, overlooked FOI issue?

41
Year 4 Replication of MF Focus on FOI

• Replicated quasi-experiment in 10 new schools (N
  2000 students).
• Teachers asked to focus on FOI.
• Wonderful teachers who understood the study goal
  study the impact of MF.

42
FOI in Year 4 with MF

• FOI included
• ISCOP (Process FOI)
• Lesson Flow (Process FOI)
• Adherence to unit (ACOP), a new instrument that
  measured close adherence to MFs structure
  (Structure FOI)
• Teacher Interviews/Surveys
• Teacher Logs
• Student Journal Entries, of responses to
  Journal (Structure FOI)
• Student survey about self-reporting use of
  instructional strategies (Process FOI)

43
Overall Results for MF in Year 4

• Student Level results using traditional ANOVA,
  ES .23
• Classroom Level results using HLM,
• ES .56 (Rethinam, Lynch, Pyke, 2008)

44
Effect Sizes for subgroups of students MF (Year
4)
45
ISCOP Strategies Means and Correlations with
Outcomes for MF
Instructional Criterion Instructional Criterion Mean (Scale 0 - 3) Mean (Scale 0 - 3) Correlation with Outcomes Correlation with Outcomes
Instructional Criterion Instructional Criterion Treatment Comparison Treatment Comparison
Identifying a Sense of Purpose Identifying a Sense of Purpose Identifying a Sense of Purpose Identifying a Sense of Purpose Identifying a Sense of Purpose Identifying a Sense of Purpose Identifying a Sense of Purpose
Conveying lesson/activity purpose 0.65 0.58 .57 -.28
Justifying lesson/activity purpose .46 .58 .56 -.40
Taking Account of Student Ideas Taking Account of Student Ideas Taking Account of Student Ideas Taking Account of Student Ideas Taking Account of Student Ideas Taking Account of Student Ideas Taking Account of Student Ideas
Assisting teacher in identifying own students ideas 2.40 1.38 -.05 -.52
Engaging Students with Relevant Phenomena Engaging Students with Relevant Phenomena
Providing variety of phenomena 1.71 1.50 .44 -.23
Providing vivid experiences 2.75 2.46 .13 .08
Developing and Using Scientific Ideas Developing and Using Scientific Ideas Developing and Using Scientific Ideas Developing and Using Scientific Ideas Developing and Using Scientific Ideas Developing and Using Scientific Ideas Developing and Using Scientific Ideas
Introducing terms meaningfully 2.48 2.38 .60 -.09
Representing ideas effectively 1.46 1.13 .16 -.01
Promoting Student Thinking about Phenomena, Experiences, Knowledge Promoting Student Thinking about Phenomena, Experiences, Knowledge
Encouraging students to explain their ideas 2.13 2.25 .62 -.20
46
ISCOP

• This generic instrument for measuring FOI
  instructional strategies did not obviously
  distinguish between MF and Comparison
  classrooms.
• But some items were correlated with student
  outcomes for MF classrooms.
• ODonnell dissertation elegantly teased out which
  instructional strategies seemed to matter for
  higher student outcomes.
• ISCOP needs more work on validity and
  reliability--OR SCALE-uP Theory of Change needs
  refinementProject 2061 Curriculum Analysis needs
  refinement?

47
Lesson Flow for Instruction for units on Motion
and Force
MF
Comparison
48
Lesson Flow

• MF classrooms provided more time for students to
  work in groups and individually than Comparison
  classrooms.
• Students construct meaning in groups, consistent
  with notion of community of practice and situated
  cognition.

49
Adhering to MF Lesson Components (ACOP)

• MF teachers adhered to MF unit gt 80 of time.
• ACOP did not predict outcomes because the range
  was narrow this was a good thing for this study
  because high fidelity in this measure of
  structure is credible and stronggood face
  validity for FOI, and highly reliablethe unit
  was well-implemented.
• Teachers adhered to the unit for this study, to
  ascertain impact of MF.
• Thank you teachers.

50
Results Teacher FOI Structure
51
Student Journals

• MF students completed their journal responses
  80 of time.
• Rates of journal question completion predicted
  classroom outcomes.

52
Unpublished Year 4 HLM Results for MF ISCOP,
ACOP, Lesson Flow, Student Journals

• HLM analysis found one classroom-level factor
  that predicted student outcomes
• --Amount of Student-Group- Centeredness
  (Lesson Flow)
• and one student level factor
• --Student Journal Completion
• Thanks to Dr. Jaewa Choi of GWU for this data
  analysis.

53
SCALE-uP Theory of Action c. 2010
Teacher Instructional Strategies
Teacher Adherence to Lesson Structure
ACOP
ISCOP
Structure FOI
Process FOI
Teacher
Curriculum Materials
Student Outcomes
Structure FOI
Student Adherence to Lessons
Student Group Centered Instruction
Process FOI
Lesson Flow
Student Journal
54
SCALE-uP Theory of Action c. 2010
Teacher Instructional Strategies
Teacher Adherence to Lesson Structure
ACOP
ISCOP
Teacher
Curriculum Materials
Student Outcomes
Student Adherence to Lessons
Student Group Centered Instruction
Lesson Flow
Student Journal
55
SCALE-uP Theory of Action c. 2010
Teacher Instructional Strategies
Teacher Adherence to Lesson Structure
ACOP
ISCOP
Teacher
Curriculum Materials
Student Outcomes
Students
Student Adherence to Lessons
Student Group Centered Instruction
Lesson Flow
Student Journal
56
SCALE-uP Theory of Action c. 2010
Teacher Instructional Strategies
ISCOP
ACOP
Teacher
Curriculum Materials
Teacher Adherence to Lesson Structure
Student Outcomes
Students
Student Adherence to Lessons
Student Group Centered Instruction
Lesson Flow
Student Journal
57
Questions?
58
Summary Findings

• Score Card CTA and MF seemed to be effective
  and equitable in this school system, Seasons did
  not seem to be effective.
• Theory of Action Changed from looking at one
  relatively generic measure of teacher process
  fidelity to multiple measures of FOI adding
  student FOI and measures of FOI structure, and
  developing a better Theory of Action.
• Lesson Flow and Student Journal response best
  predicted student outcomes ISCOP analysis
  illuminated most potent instructional strategies
  for MF.

59
FOI Lessons Learned

• Studying curriculum adaptations for an
  unproven curriculum is a slippery slope because
  critical ingredients are unknown.
• Study of FOI in Comparison groups provides
  important counterfactual.
• This study shows potential for both teacher and
  student FOI, as well as process and structural
  approaches.
• Study of FOI should reveal more about critical
  components of an intervention if measures of each
  component shows a positive relationship with
  student outcomes.

60
The End

• Thanks to Researchers Without Borders for
  sponsoring this Webinar.

61
References

• American Association for the Advancement of
  Science (AAAS). (2003). Project 2061 middle
  grades science textbooks A Benchmarks-based
  evaluation. Retrieved June 1, 2004, from
  http//www.project2061.org/tools/textbook/mgsci/in
  dex.htm.
• Dusenbury, L., Brannigan, R., Falco, M.,
  Hansen, W. B. (2003). A review of research on
  fidelity of implementation Implications for drug
  abuse prevention in school settings. Health
  Education Research Theory and Practice, 18(2),
  237-256.
• Harvard-Smithsonian Center for Astrophysics.
  (2001). ARIES Exploring motion and forces
  Speed, acceleration, and friction. Watertown, MA
  Charlesbridge Publishing
• Kesidou, S., Roseman, J.E. (2002). How well do
  middle school science programs measure up?
  Findings from Project 2061s curriculum review.
  Journal of Research in Science Teaching, 39(6),
  p. 522-549
• Lastica, J.R., O'Donnell, C.L. (2007, April).
  Considering the role of fidelity of
  implementation in science education research
  Fidelity as teacher and student adherence to
  structure. In C. O'Donnell (Chair), Analyzing the
  relationship between Fidelity of Implementation
  (FOI) and student outcomes in a quasi-experiment.
  Symposium conducted at the Annual Meeting of the
  American Educational Research Association,
  Chicago, IL.Lawrence Hall of Science.
  (2000).GEMS The real reasons for
  seasonsSun-Earth connections. Berkeley The
  Regents of the University of California.
• Lynch, S. (2000). Equity and science education
  reform. Mahwah, NJ Lawrence Erlbaum and
  Associates.
• Lynch, S., Kuipers, J.C., Pyke, C., Szesze, M.
  (2005). Examining the effects of a highly rated
  science curriculum unit on diverse students
  Results from a planning grant. Journal of
  Research in Science Teaching, 42(8), 912-946.

62

• Lynch, S. ODonnell, C. (2005, April). The
  evolving definition, measurement, and
  conceptualization of fidelity of implementation
  in scale-up of highly rated science curriculum
  units in diverse middle schools. In S. Lynch
  (Chair), The role of fidelity of implementation
  in quasi-experimental and experimental research
  designs Applications in four studies of
  innovative science curriculum materials and
  diverse student populations. Symposium conducted
  at the Annual Meeting of the American Educational
  Researchers Association, Montreal, Canada.
• Lynch, S., O'Donnell, C., Hatchuel, E.,
  Rethinam, V. (2007, April). A model predicting
  student outcomes in middle school science
  classrooms implementing a highly-rated science
  curriculum unit. Paper presented at the Annual
  Meeting of the National Association for Research
  in Science Teaching, New Orleans, LA.
• Lynch, S., O'Donnell, C., Hatchuel, E., Rethinam,
  V., Merchlinsky, S., Watson, W. (2006, April).
  Whats up with the Comparison group? How large
  quasi-experimental study of highly rated science
  curriculum units came to grips with unexpected
  results. Paper presented at the Annual Meeting of
  the American Educational Research Association,
  San Francisco, CA.
• Lynch, S., Taymans, J. Watson, W., Ochsendorf,
  R., Pyke, C. Szesze, M. (2007). Effectiveness
  of a highly-rated science curriculum unit for
  students with disabilities in general education
  classrooms. Exceptional Children, 73(2), 202-223.
• Merchlinsky, S. Hansen-Grafton, B. (2007,
  April). Considering the role of Fidelity of
  Implementation (FOI) in science education
  research Evaluation and science specialists'
  role in collecting FOI data in a large school
  district. In C.L.
• Michigan Department of Education. (1993).
  Chemistry That Applies. The State of Michigan.
• Mowbray, C., Holter, M. C., Teague, G. B.,
  Bybee, D. (2003). Fidelity criteria Development,
  measurement, and validation. American Journal of
  Evaluation, 24(3),315-340.
• ODonnell, C. L. (2007). Fidelity of
  implementation to instructional strategies as a
  moderator of curriculum unit effectiveness in a
  large-scale middle school science experiment.
  Dissertation Abstracts International, 68(08).
  (UMI No. AAT 3276564)

63

• O'Donnell, C.L., Lynch, S., Lastica, J.,
  Merchlinsky, S. (2007, April). Analyzing the
  relationship between Fidelity of Implementation
  (FOI) and student outcomes in a quasi-experiment.
  Symposium conducted at the Annual Meeting of the
  American Educational Research Association,
  Chicago, IL.
• O'Donnell, C.L., Lynch, S., Watson, W.,
  Rethinam, V. (2007, April). Teacher and student
  Fidelity of Implementation (FOI) to process
  Quality of delivery and student responsiveness
  and relationships to classroom achievement. In
  C.L. O'Donnell (Chair), Analyzing the
  relationship between Fidelity of Implementation
  (FOI) and student outcomes in a quasi-experiment.
  Symposium conducted at the Annual Meeting of the
  American Educational Research Association,
  Chicago, IL.
• O'Donnell, C. (2008). Defining, conceptualizing,
  and measuring fidelity of implementation and its
  relationship to outcomes in K-12 curriculum
  intervention research. Review of Educational
  Research, 78(1), 33-84.
• O'Donnell, C., Lynch, S. (2008, March).
  Fidelity of implementation to instructional
  strategies as a moderator of science curriculum
  unit effectiveness. Paper presented at the 2008
  annual meeting of the American Educational
  Research Association, New York, NY.
• Rethinam, V., Pyke, C., Lynch, S. (2008). Using
  multilevel analyses to study the effectiveness of
  science curriculum materials. Evaluation and
  Research in Education, 21(1), 18-42

64

• Rethinam, V., Pyke, C., Lynch, S. (in
  preparation.). Using Multilevel Analyses to Study
  Individual and Classroom Factors in Science
  Curriculum Effectiveness.
• Songer, N. B., Gotwals, A. W. (2005, April).
  Fidelity of implementation in three sequential
  curricular units. In S. Lynch (Chair), Fidelity
  of implementation in implementation and scale-up
  research designs Applications from four studies
  of innovative science curriculum materials and
  diverse populations. Symposium conducted at the
  meeting of the Annual Meeting of the American
  Educational Research Association. Montreal,
  Canada.
• Stern, L. Ahlgren, A. (2002). Analysis of
  students assessments in middle school curriculum
  materials Aiming precisely at benchmarks and
  standards. Journal of Research in Science
  Teaching, 39, 889-910.
• Watson, W., Lynch, S., Rethinam, V., ODonnell,
  C. (2006, April). Development of an instrument
  to measure student responsiveness to
  implementation of science curriculum materials.
  Paper given at the annual meeting of the National
  Association for Research in Science Teaching,
  April, 2006 , San Francisco.
• Further acknowledgments of important
  contributions to this work on fidelity of
  implementation made by Rob Ochsendorf, Aiyita
  Ruiz-Primo, Doug Clement, Okhee Lee, Bruce Ward,
  Carolyn Walton, Theron Blakeslee, Andy Anderson,
  Phyllis Blumenfeld.