Challenges of Conducting Research in the Real World

1
Challenges of Conducting Research in the Real
World
Julie Gess-Newsome, NAU Janet Carlson,
BSCS NSF DRK12 PI Meeting, November 2008
2
Session Purpose

• Provide an overview of the tentative findings
  from Project PRIME
• Lead a discussion concerning the challenges of
  conducting educational research in schools over
  time
• Illustrate challenges encountered in Project
  PRIME
• Through group discussion, generate potential
  solutions to common challenges
• Contextual Issues
• Recruitment and Retention
• Research Design
• Quality of Data

3
Promoting Reform through Instructional Materials
that Educate (PRIME)

• How does teacher PCK and student achievement
  change with the use of
• Educative curriculum materials on their own?
• Combined with professional development?
• Two year intervention with 40 biology teachers
• Teachers acted as the unit of analysis
• Teacher and student data baseline -gt
• curriculum materials OR
• curriculum materials and professional
  development -gt
• teaching-gt
• PD or teaching again

4
Measurements

• PCK Reflections (CK, PK, CxK, PCK)
• Structured Interviews (CK, PK, CxK, PCK)
• Videotapes of Classroom Instruction (CK, PK, CxK,
  PCK, Fidelity)
• Major Field Test for Biology (ETS) (CK)
• Reform Teaching Observation Protocol (CK, PK,
  CxK)
• Classroom Observation Checklist (PK)
• Student Achievement
• Student Work
• Professional Learning Community Rating Scale

5
Tentative Finding 1. Student achievement scores
increased significantly from pretest to posttest.
(N527)
6
Tentative Finding 2. Teacher MFTB scores showed
inconsistent gains over time. (N17)
7
Tentative Finding 3. There is a positive
correlation between teacher scores on the MFTB
and student gain scores in content achievement.
There is a slight though positive correlation
between teacher MFTB scores and RTOP scores.
8
Tentative Finding 4 Teachers PCK trajectories
vary by individual, sometimes unpredictably
9
Tentative Finding 5 The relationship of data
across a single teacher is unclear.
10
Challenges of Conducting Educational Research

• Contextual Issues
• Recruitment and Retention
• Research Design
• Quality of Data

11
Contextual Issue 1 Lag time between design
implementation

• Biology

• Integrated Science

12
Contextual Issue 2 Mismatched Conceptions
Whats the best curriculum for our biology
program?
Which program will best support our teachers in
their science teaching?
How do I know if my students are learning?
13
Contextual Issue 3 Views of Curriculum

• Implementer

• Designer

14
Recruitment and Retention 1 Time
15
Recruitment and Retention 2 Incentives vs.
Commitments
Money Graduate Credits Professional
Development Curriculum Materials Stipends Travel
Support Conference Support
Teacher Data Videotapes Student Data Time on
Campus Reflections Interviews
16
Recruitment and Retention 3 Participant
Attrition
17
Research Design 1 Random Assignment
I bet we get selected for the experimental group!
18
Research Design 2 Statistical Significance
We Want In!
19
Quality of Data1 Researcher vs. Teacher
Expectations
20
Quality of Data 4 Researcher vs. Student
Expectations
21
Quality of Data3 Data Fatigue

22
Quality of Data4 Instrumentation

23
Quality of Data 5 IRB and Informed Consent
24
Quality of Data 6 EquipmentLoaned or
Owned
25
Solution Generation

• How typical are these experiences to those that
  you have encountered in your research?
• What solutions have you found to help with these
  challenges?
• Please record potential solutions to report out
  to the group.

26
Thank you

• For further information contact the authors
• Janet Carlson jcarlson_at_bscs.org
• Julie Gess-Newsome julie.gess-newsome_at_nau.edu
• A full copy of the research paper about Project
  PRIME can be found at
• http//bscs.org/NARST08