Evaluation on a Shoestring

1
Evaluation on a Shoestring

• Participatory Techniques for the Evaluator with
  Minimal Funding

2
Overall goals for the morning

• Learn how to use 11 interactive techniques
• Study frameworks for participatory evaluation
  practice
• Discuss how to overcome barriers to participatory
  evaluation
• Reflect on how to apply these ideas to your own
  evaluation practice

3
The mornings agenda

• Seven techniques before the break
• Lecturette Frameworks and psychological
  principles
• Four more techniques before lunch
• Throughout the morning Continuing
  reflection on your own practice

4
Types of techniques

• For responding to set content
• (1-5)
• For generating information
• (6-10)
• For organizing information
• (11)

5
The basic tasks of inquiry

• Framing questions
• Determining an appropriate design
• Identifying a sample
• Collecting data
• Analyzing data and presenting results
• Interpreting results
• Reporting

6
Important point

• You can use
• these techniques THROUGHOUT
• the evaluation process
• Not just at the beginning. . .

7
Why use participatory techniques?

• How can participatory techniques help evaluators?

8
My participatory principles

• Building peoples capacity to think evaluatively
  matters.
• Participation in evaluations should be a learning
  experience.
• It is essential to involve people actively in
  evaluations.

9
How participation helps

• People invest in the evaluation process and
  outcomes
• It makes evaluation less scary
• Some will learn evaluation skills

It is fun!
10
Technique 3- Statement 1

• State specialists should include an evaluation
  tool for each program they introduce to county
  staff.
• Strongly Disagree Agree Strongly
• Disagree Agree

11
Technique 3- Statement 2

• State specialists should be responsible for
  compiling overall evaluation results that will
  lead to the creation of impact statements for ND
  reporting.
• Strongly Disagree Agree Strongly
• Disagree Agree

12
Technique 3- Statement 3

• County staff should be required to enter all
  county survey results into a system that will
  allow state staff to aggregate data and produce
  impact reports.
• Strongly Disagree Agree Strongly
• Disagree Agree

13
Debrief strategies for responding to set content

• 1- Voicing variables
• 2- Fist-to-five
• 3- Belief sheet
• 4- Dot voting
• 5- Corners

14
Strategies for generating information

• 6- Three-step interview
• 7- Data dialogue
• 8- Making metaphors
• 9- Check-in
• 10- Graffiti/carousel

15
Technique 6- Three-step interview

• Three roles create three steps
• Interviewer
• Interviewee/respondent
• Recorder
• The interview process is structured to build on
  social psychological principles of cooperation

16
Technique 7- Data dialogue

• A process to use when you cannot afford focus
  groups
• It takes advantage of some of the processes of
  the three-step interview
• Can be useful in community settings

17
Analysis exercise

• In what ways is a data dialogue like a three-step
  interview?
• In what ways is it different?

18
Conceptual frameworks for participatory techniques

• Useful for planning
• Helpful for analysis

19
What roles can evaluators play?

• A relationship exists between the evaluator and
  the client, the program staff, and other
  evaluation stakeholders
• The evaluation decision-making and implementation
  relationship may shift during the study

20
Interactive Evaluation Quotient
21
Examples of evaluator roles

• Technical expert on research design, measurement,
  statistics
• Facilitator of group interaction
• Coach of others doing their own evaluations
• Others?

22
Types of participant involvement

• Mere awareness
• Passive support or minimal participation
• Active participation in the evaluation process
• Commitment to consider and ultimately use the
  evaluation results

23
Social interdependence theory

• Goal
• Structures Interactions Outcomes

24
Social interdependence theory

• At best
• Cooperative goal structures (positive
  interdependence)
• Promotive (responsive) interaction
• Constructive outcomes
• At worst
• Competitive goal structures (negative
  interdependence)
• Oppositional (obstructive) interaction
• Destructive outcomes

25
Structuring cooperative participation

• Positive interdependence
• Individual accountability
• Promotive (face-to-face) interaction
• Social skills
• Group processing
• (Adapted from Johnson Johnson, 2000)

26
Structuring positive interdependence

• Identify a common purpose
• Create shared benefits or consequences
• Provide one set of materials
• Assign complementary and interconnected roles
• . . . plus

27
Structuring positive interdependence

• Designate an outside force to motivate people to
  coordinate efforts
• Arrange the workspace purposefully
• Have the group establish a shared identity

28
Strategies for generating information

• 6- Three-step interview
• 7- Data dialogue
• 8- Making metaphors
• 9- Check-in
• 10- Graffiti/carousel

29
Strategy for organizing information

• 11- Concept
• formation

30
Techniques 10 and 11

• Can be done on the wall, informally, and is then
  called graffiti
• Can be done on flipchart paper passed among
  groups and is then called carousel
• Always coupled with Technique 11- Concept
  formation

31
Technique 8- Making metaphors

• A is worth 1000 words
• tall impressive
• BIG magnificent terrifying
• High Not like home awesome sun struck

32
Technique 9- Check-in

• What is the most important idea you have learned
  this morning?

33
Thank you, and good luck!

• Jean A. King
• kingx004_at_umn.edu