Organizational Design, Diagnosis, and Development

1
Organizational Design, Diagnosis, and Development

• Session 15
• The Evaluation of Change

2
Objectives

• To understand issues dealing with the evaluation
  of change
• To consider some of the major problems in
  evaluating change programs
• To review designs for evaluation

3
Where Evaluation Fits
Implementation of Intervention
Diagnosis
Design of interventions
Implementation Feedback
Evaluation Feedback
4
EVALUATING OD INTERVENTIONS

• Evaluation to guide intervention -- formative
  evaluation
• Is it reaching right audience?
• Is the audience realizing some change?
• Summative evaluation -- Impact on the
  organization
• Is the organization realizing the effect it
  desired?

5
Problems in Evaluating Change

• Vague goals
• Strong promises
• Weak effects
• Other problems
• Finding an appropriate criterion
• Finding skilled personnel
• The politics of evaluation
• When to evaluate

6
Levels of Evaluation

• Reaction
• Learning
• Behavior
• Results

7
Questions to be Asked

• Does examination of the criteria indicate change
  has occurred?
• Can the change be attributed to the intervention?
• Is it likely that new participants in the program
  will demonstrate similar changes?
• Is it likely that new participants in the same
  program will demonstrate similar changes in a
  different site?

8
Pre-experimental Designs

• One group post-test only
• I (intervention) T (evaluation)
• One group pretest/post-test design
• T-1 (pre-test) I T-2 (post-test)

9
Experimental Designs

• Pretest/post-test control-group design
• E T-1 (pre-test) I T-2 (post-test)
• C T-1 (pre-test) T-2 (post-test)
• Solomon four-group design
• G1T-1 (pre-test) I T-2 (post-test)
• G2T-1 (pre-test) T-2 (post-test)
• G3 I T-2 (post-test
• G4 T-2
  (post-test)

10
Quasi-Experimental designs

• Time series
• T1 T2 T3 T4 I T5 T6 T7 T8
• Non-equivalent control group
• E T-1 (pre-test) I T-2 (post-test)
• C T-1 (pre-test) T-2 (post-test)

11
Utility Analysis

• Cost benefit analysis
• Cost effectiveness analysis
• An example for absenteeism program
• document the rate of absenteeism
• calculate absenteeism hours lost per year
• calculate absenteeism compensation costs
• calculate cost of managing absenteeism
• calculate other costs of absenteeism

12
Threats to Evaluation Validity

• Weak evaluation designs
• Select groups for introducing intervention

13
Backwards Forwards

• Summing up In todays session we examined some
  problems that surround evaluation, looked at
  several designs evaluation intervention
  (pre-experimental, experimental, and quasi
  experimental designs). Threats to evaluation
  validity were also discussed
• Looking ahead Next time we begin our examination
  of human process interventions with a look at
  interpersonal and group process approaches