Benchmarking and Evaluation: Toward Results Driven Professional Development

1
Benchmarking and Evaluation Toward Results
Driven Professional Development

• Kirk Vandersall
• Managing Director
• www.arroyoresearchservices.com
• kirk_at_arroyoresearchservices.com

2
If we knew what it was we were doing, it would
not be called research, would it? Albert
Einstein.
3
The Plan

• Relationship of Benchmarking and Evaluation
• Results Driven Professional Development
• Knowing What Youre Doing
• Seeing Your Progress
• Evaluation Models and APQC Data
• Practical Considerations for Model Driven
  Evaluation
• Questions

4
Benchmarking and Evaluation

• Role of Benchmarking Data
• When are Benchmarks Outcomes?
• How do Benchmarks Drive Outcomes?
• Whats a Relevant Benchmark?

5
Results Driven Professional Development

• Who, what, when, how
• To each their owngoals
• To each their ownprocess
• To each their ownoutcomes

6
Driving Results Through Evaluation

• Discover
• Envision
• Articulate
• Align
• Measure
• Evaluate

7
(No Transcript)
8
(No Transcript)
9
Knowing What Youre Doing
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
Survey of Enacted Curriculum
14
Models Revisited Seeing Your Progress

• Organize disparate ideas and data
• Identify key data points
• Show progress toward major goals
• Identify important intermediate steps

15
Absolute certainty is a privilege of uneducated
minds - and fanatics. It is, for scientific folk,
an unattainable ideal. Cassius J. Keyser
16
If Model Driven

• Sufficient intervention intensity
• Multifaceted intervention design
• Plan for handling multi-level effects
• Plan for isolating the effects of the intervention

17
Models

• NSDC
• Guskey
• PCMM
• APQC
• Hybrid

18
NSDC Model

• Awareness of Information
• Deep understanding of information
• Capability for implementation
• Mechanical implementation
• Refined and Routine Implementation

19
Awareness of Information

• What Initial introduction to new material,
  including pedagogy, content, curriculum,
  programs, or other educational methods or
  innovations
• How Materials distribution, Meetings, Workshops,
  Courses, Peer introduction

20
Deep Understanding of Information

• What Work with new information in multiple ways,
  through appropriate adult learning processes,
  over reasonable time periods. Sometimes includes
  writing new models, course modules, papers, or
  other teacher work products.

21
Deep Understanding How

• PD courses
• PD programs/initiatives
• Mentoring and peer coaching
• Simulations
• Portfolio development
• Teacher work samples
• PLC participation

22
Capability for Implementation

• What Acquire necessary new skills to apply new
  knowledge in the classroom practice new methods
  or approaches outside the classroom

23
Capability for Implementation How

• Skills analysis (what skills are needed for the
  new approach)
• Skills assessment or inventory
• Modeling and examples of best practice
• Simulations
• Practice sessions
• Mentoring and coaching

24
Mechanical implementation

• What Begin to use new knowledge, approaches and
  skills by the book, consciously going through
  the steps provided in PD or resource materials

25
Mechanical Implementation How

• Implementation guides
• Classroom resources (student handouts,
  appropriate materials)
• Lesson plans
• Coaching and modeling
• Monitoring
• Examples of how others in the school/district
  have done it

26
Refined and Routine Implementation

• What Teachers incorporate new
  methods/approaches/skills with the rest of their
  teaching repertoire in a way that assures the new
  teaching approaches are authentic, automatic, and
  appropriate. i.e. refined and routine.

27
Refined and Routine Implementation How

• Integration of new ideas and approaches with
  curriculum
• Implementation guides
• Classroom resources (student handouts,
  appropriate materials)
• Lesson plans
• Coaching and modeling
• Monitoring
• Examples of how others in the school/district
  have done it

28
Guskey Model
29
Level 1 Participants' Reactions
30
Level 2 Participants' Learning
31
Level 3 Organization Support and Change
32
Level 4 Participants' Use of New Knowledge and
Skills
33
Level 5 Student Learning Outcomes
34
Capability Maturity Model
35
Climbing the Capability Maturity Model
36
(No Transcript)
37
Practical Considerations

• Data sources
• Data management
• Immediate, intermediate and long-term measures
• Data and evaluation use
• Target audiences
• Frequency of measurement and data use
• Research design, research design, research design

38
You dont need an F-test to know that theres a
trout in your milk.
39
(No Transcript)
40
Pre-Post Control Group
N O X O N O O

• Example Teacher are assigned or selected to an
  intervention by school, but have not been
  randomly assigned to schools

41
Regression Discontinuity
C O X O C O O

• K12 Example Teachers are assigned to
  professional development experiences based on
  assessment scores

42
Proxy Pretest Design
N O1 X O2 N O1 O2

• K12 Example Outcome of interest is a teaching
  skill to be measured at the end of the project
  prior year assessment of related content
  knowledge is available

43
Switching Replications Design
N O X O O N O O X O

• K12 Example ½ of middle school teachers
  participate in first year, other ½ in second year

44
Not everything that can be counted, counts not
everything that counts can be counted.
Albert Einstein
45
Arroyo Research Services

• multi-disciplinary
• cross-functional
• user focused
• action oriented
• standards-based
• policy wise

46
Arroyo Research Services

• Program Evaluation
• Consulting and Professional Services
• Data Collection and Analysis
• Scientifically Based Research
• Project Design and Proposal Writing
• Application Development
• Workshops, seminars, and speaking