Empirical Study of the Effects of Professional Development on Improving Mathematics and Science Inst

1
Empirical Study of the Effects of Professional
Development on Improving Mathematics and Science
InstructionMSP Goal 3 NSF/RETA Study Website
http//www.ccsso.org/msppdstudyPI, Rolf K.
Blank, rolfb_at_ccsso.org Council of Chief State
School Officers Co-PI, Kwang Suk Yoon, Bea
Birman, Mike Garetksyoon_at_air.org American
Institutes for ResearchCo-PI, John Smithson,
johns_at_education.wisc.eduWisconsin Center for
Education Research
2
Demonstration Project to show
how multiple PD programs can be evaluated using
a common set of research-based measures of quality
and
how a survey methodology can be used to evaluate
the effects of PD in improving math science
instruction
3
An evaluation study to determine
1) whether PD activities supported by MSP
programs are consistent with research-based
definitions of quality PD
2) The effects of PD on mathematics/ science
instructional practices and content
3) How MSP programs use study findings to improve
PD effectiveness
4
A Logic Model for Assessing the Effect of
Professional Development on Instructional
Practice and Content Alignment
Quality of PD Content Focus Collective
Participation Active Learning Coherence Sustain
ed Effort
Teacher Characteristics Background
Variables, Prior PD Experiences Target Class
Students Diversity School Culture Trust
PDAL
Pre-Instruction Alignment of Instruction with
Content Standards Instructional Practice
Post-Instruction Alignment of Instruction with
Content Standards Instructional Practice
SEC1
SEC2
Year 1
Year 2
Year 3
Year 0
5
Study Sample

• MSP Sites (N 5 Cleveland, New Jersey,
  Brockport-SUNY, El Paso, Corpus Christi, TX)
• Voluntary participation, Data feedback to sites
• School Districts (N 23)
• Urban/Rural, Size varies
• Schools (N 79)
• Treatment vs. Control (based on MSP PD cohort)
• Teachers (N 570)
• Middle Grades Math and Science
• Target Class Data analysis one class/teacher

6
SEC Data Instruments
Instructional Content Topics by Cognitive
Demand
Instructional Practices Teacher Characteristics
Multi- purpose indicators of curriculum practice
and professional development opportunities.
7
SEC-MSP/PD Data Instruments
Instructional Content Topics by Cognitive
Demand
Instructional Practices Teacher Characteristics
P D A L
8
(No Transcript)
9
Classroom Practices
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
New Improved !
Classroom Practices
14
After
PDAL
Before
15
Schools
PDAL
Teachers
16
Year 1 Data Collection Timeline
April 2003 May 2003 June 2003
Survey orientation Teacher Incentives Admin.
Training Survey Administration Site
Coordinators Group Admin.
P D Activity Logs
17
Teacher Incentives
Review Reflection Planning

• On-line access to survey results

• Secure access to individual results

• Compare instruction across grades

• Compare instruction standards

• Honoraria

18
Survey Administration

• Group Administration

• 60 to 90 minutes

• Brief teachers

• Back cover information
• Voluntary (return blank)
• Confidential (seal envelope)
• Target Class / Reporting period
• IP instructions / Response metric
• Content instructions
• Feedback forms

• Survey Return Procedures

19
Survey Back Cover Respondent Information
20
Instructions for Selecting the Target Class


Mathematics
Instruction -- For all questions about classroom
practices please refer only to activities related
to mathematics instruction. If you teach more
than one mathematics class, select the first
class that you teach each week. If you teach a
split class (i.e. the class is split into more
than one group for mathematics instruction)
select only one group to describe as the target
class.
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
Professional Development Activity Log A New
Approach to Design, Measurement, Data Collection,
and Analysis
PDAL Data Analysis Plan December 11, 200
Kwang Suk Yoon American Institutes for Research
To be presented at AERA 2004 Annual Meeting
28
The purpose of the study is to determine
Assessing the Effects of Professional Development
on Improving Mathematics and Science Instruction

• Whether professional development (PD) activities
  supported by MSP programs are consistent with
  research-based qualities of PD
• The effects of PD on mathematics/science
  content alignment and instructional practice

29
Limitations of Existing Instruments for
Collecting PD Data

• Rely on teachers retrospective reports on
  cumulative professional development experiences
  that have occurred over an extended time period
• Teachers may have to aggregate across a number of
  distinct activities
• Focus on a particular set of PD activities
  supported by certain programs (e.g., Eisenhower)
  and may exclude many others
• Solicit from teachers their opinions about their
  PD activities rather than their actual behavioral
  engagement with the activities

30
Features of Professional Development Activities
Log (PDAL)

• Help teachers create an ongoing monthly log of
  any professional learning activity in which they
  participate
• Web-based, self-administered survey
• Longitudinal design Data collected over 15
  months
• Aligned with SEC items (e.g., content focus)
• Inclusive approach to PD
• Includes MSP-sponsored and non-MSP-sponsored
  activities
• Documents one-time and recurring activities
• Captures formal and informal professional
  development activities

31
PDAL Entries

• Name of activity
• Number of hours spent on each activity and its
  duration
• Whether the activity is a one-time event or a
  continuous one (i.e., recurring over a number of
  months)
• Type of activity (e.g., workshop, summer
  institute, study group)
• Purpose of activity (e.g., strengthening subject
  matter knowledge)
• Content focus (e.g., algebraic concepts absolute
  values, use of variables, etc.)
• Instructional practice instructional topics
  covered in each activity (e.g., use of
  calculators, computers, or other educational
  technology)
• PD quality features (e.g., active learning,
  coherence, collective participation)
• Materials used during each activity

32
Advantages of PDAL

• Collects disaggregate information about specific
  PD activities Reduces bias introduced by gross
  data aggregation
• Gathers accurate, time-sensitive information
  Minimizes recall problem with retrospective
  reports
• Allows teachers to review their own logs
  Teachers can reflect on their own PD experiences
• Generates context sensitive questions
• Alleviates teachers response burden
• Minimizes data entry errors
• Be able to tailor technical assistance to
  teachers based on their response patterns

33
Obstacles to PDAL

• Requires users technical knowledge skills
• In the use of computer and web
• In the specific features of the PDAL
• Solution Provide Technical Assistance (TA) as
  needed
• Requires Web access
• Solution No-web-access is considered as out of
  scope
• The length of data collection period and
  teachers willingness
• Sample attrition
• Frequency of reporting
• Missing data
• Solution Solicit teachers cooperation through
  reminder calls and postcards and local MSP
  project leaders initiatives

34
Sample and Multi-level Contexts

• MSP Sites (N4-5)
• Treatment vs. Control, Nature of partnership
• School Districts (N16-23)
• Content Standards, Urban/Rural, Size
• Schools (N67-79)
• SES, School Culture Trust
• Teachers (N450-570)
• Subject, Gender, Teaching Experience, Credential
• Target Class
• Content Alignment Content Match Student
  Diversity (Minority, Achievement Levels)

35
Hypothetical PDAL DataConfigurations and
Aggregation

• Disaggregate log entry data
• Monthly log (as a basic unit of observation)
• Teacher
• Activity
• Time
• Data aggregation
• Time
• Activities
• Teachers
• Sites

36
Descriptive Analysis

• Descriptions correlates of PD activities
• patterns of responses (logs)
• Timing of log entries (peak and low) average
  time per log
• Patterns of PD activities
• Average number of PD activities per teacher
  seasonal variation in PD activities (overall
  participation rate, rate by type of activity)
  mean quality by site or PD type
• Correlates
• Which types of PD activities are associated with
  high quality
• Classes of teachers
• Some latent classes of teachers (e.g.,
  minimalists, enthusiasts, informal learners)

37
Impact Analysis

• Main Outcome Variables of PDAL Analysis
• Quantity and Quality of PD
• Main Outcome Variables of SEC Analysis
• Change or Stability in Content Alignment
• Change or Stability in Instructional Practice
• Linking PD Variables to Instructional Outcomes

38
Impact Analysis (Contd)

• Main Effects of
• MSP Site
• Treatment
• PD Quality
• School Culture (e.g., Trust)
• Subject
• Interaction Effects of
• Treatment by MSP Site
• Treatment by PD Quality
• Treatment by School Culture
• Treatment by Subject
• Treatment by PD Quality by School Culture

39
Example of Interaction Effect The Effect of
Treatment on the Use of Instructional Practice
Moderated by School Culture (Trust)
40
Effects of Professional Development on Change in
Teachers Instructional Practice By PD
Activitys Focus on Specific Instructional
Strategy, PD Quality, and School Culture