Targeted Evaluation Process

1
Targeted Evaluation Process

• Michael E. Gallery, PhD, CAE
• President
• OPIS, LLC
• mgallery_at_opisconsultants.com
• 630 Sugarloaf Ct.
• Highland Village, TX 75077
• 469-235-5165

2
OPIS

• Helping to turn ideas into action

3
Performance Improvement Model
4
The Purpose of Evaluation

• To prove
• To Improve

5
Reasons for avoiding Evaluation

• Time commitment
• Cost
• Fear
• Political climate

6
The fact is

• Everything will be evaluated
• The questions that remain are
• How? Buy whom? By what standard

7
Basic Concepts in Evaluation

• Formative
• Summative
• Reliability
• Validity

8
Formative Evaluation

• Purpose is to improve
• Done early and often
• Procedures less formal

9
Summative Evaluation

• Purpose is to prove
• More formal
• Occurs at the end.

10
Reliability

• A measure of error
• Error typically from three sources
• Time
• Sample of items
• Sample of people

11
Validity

• A measure of the evaluation tools relevance.
• Answers whether the tool measures what it
  purports to measure
• Types
• Content
• Concurrent
• Construct
• Predictive

12
Three major access points

• Front-end (needs assessment)
• Intervention prototype
• Post-intervention

13
Evaluation guides decisions

• Be clear about what decisions you will make from
  the data
• Make sure your decisions/questions can be
  measured
• Secure Collaboration
• Agree on criteria

14
Basic Elements of a System

• The behavior of each element has an effect on the
  behavior of the whole.
• The behavior of the elements and their effects on
  the whole are interdependent
• None of the elements has an independent effect on
  the whole

15
Evaluation A systems view

• Evaluation is a process of collecting information
  and feeding it back to those who need the
  information so that the system can succeed
• (Shrock Geis, 1999)

16
Performance Improvement Model
17
Evaluating Business Needs

• Objectives should emanate from documented member
  needs
• Objectives should be stated in measurable terms

18
Properly stated objectives

• Not Enhance membership
• But Increase active membership by 3
• Not increase customer service
• But The average customer rating will be at least
  4.0

19
Evaluating Business Needs

• Measure your present position
• (Data should relate directly to your objectives)
• Asses gap between is and should
• Can you live with it?
• Will the return for closing the gap exceed the
  cost of leaving it alone?

20
Evaluate the Gap

• Does the movement from is to should
  represent
• Safe bet?
• Stretch?
• Pipedream?

21
Performance Analysis

• What will people (both members and staff) need to
  do in order to achieve the objective?
• Develop a performance model

22
Performance Defined

• Performance action outcome
• Not collects membership needs data
• But uses needs data to identify meeting topics
• Not follows time management techniques
• But uses effective time management skills to
  accomplish performance objectives

23
Measure Performance Gaps

• Performance measures should be based on
  performance models
• Measure behavior, not personality traits

24
Organizational Systems Evaluation

• All human systems are open requiring outside
  energy and feedback.
• Member feedback?
• Environmental Input?
• What outside influences could affect
  organizational performance?

25
Competing Subsystems Drain Energy from the Whole.

• All subsystems working toward the same end?
• One subsystem dominating another?
• Too focused on perfection?

26
Successful Systems Adapt

• Feedback loop connected with receivers?
• Are ineffective subsystems removed?
• Changes to subsystems based on data?

27
Effective Systems React Intelligently to the
Environment

• Ongoing environmental analysis tied to the
  organization
• Build on current and accurate data

28
Focus on One Subsystem at a Time

• Trying to be all things to all people?
• Know what NOT to do?
• Fix one problem at a time?
• Does a subsystem fix relate to performance of the
  whole?

29
Organizational Process

• Subsystem Output properly feed other subsystems?
• The whole?
• Process maps exist?

30
Organizational Culture

• Are behavioral patterns and norms aligned with
  organizational intent?
• Do subsystems have organizational buy-in?

31
Organizational Infrastructure

• Do organizational mechanisms support business
  intent?
• Behavior required by subsystems should
  achievement of business intent easier than if
  subsystem were not in place

32
Subsystem Evaluation

• What are the real (actual) outputs?
• How important are these outputs to members of the
  group?
• How important are the outputs to achieving the
  business intent?
• What is obstructing the delivery of outputs?
• Can the obstructions be removed/managed?

33
Reward and Incentive Systems

• Pay tied to performance?
• Individual performance tied to business intent?
• Pay tied to outcomes or process?

34
Reward and Incentive Systems

• SMART Principle
• Specific
• Meaningful
• Achievable
• Reliable
• Timely

35
Job Aides

• Typically a written guide/instructions
• Used for infrequent/unfamiliar tasks
• Highly applicable for volunteers

36
Process Improvement

• Support business needs?
• Process maps?
• Stakeholder buy-in
• Customer feedback?

37
Process Improvement Options

• Elimination
• Re-sequencing
• Streamlining approval
• Standardization
• Use of common terminology (user driven)7

38
What Do You Plan To Do?

• What do you plan to evaluate when you get home
  What decisions will you make?
• Who will you involve?
• What is your plan?

39
Suggested Readings

• Brethower, D. General systems theory and
  behavioral psychology In Stolovitch H Keeps
  (Eds) Handbook of Human Performance Technology.
  San Francisco Jossey-Bass/Pefiffer, 1999
• Combs W Falletta S The Target Evaluation
  Process. Washington DC. American Society for
  training and Development. 2000
• Robinson D Robinson J Performance Consulting
  San Francisco Berrett-Koehler Publishers 1995
• Shrock S, Geis G. Evaluation. In Stolovitch H
  Keeps (Eds) Handbook of Human Performance
  Technology. San Francisco Jossey-Bass/Pefiffer,
  1999