Thank You for turning off cell phones

1
Thank You for turning off cell phones

• gt Its good for us!

2
Excelling at Practical Program Evaluation

• gt And loving it!

Joseph L. Donaldson Extension Specialist
3
Today we will

• Improve evaluation techniques
• Match response scales to outcomes
• Use two tools for post-then-pre-test
• Explore an online survey builder approach

4
What is evaluation?

• Think of evaluation as asking questions for the
  purpose of learning about our program so we can
  continually improve and communicate program
  outcomes to our clients and stakeholders.

5
What should I evaluate?

• Planned
• Most contact hours since thats where we have the
  most impact
• Likely to be offered again
• Measurable outcomes
• Important to your stakeholders

6
Evaluation Plans

• Watch them Observation
• Examine what they did or said Records
• Ask them Interviews or questionnaires

7
Improve Evaluation Techniques

• Observation
• Records
• Interviews and Questionnaires

8
Observation

• What are you aiming for?
• Have criteria written down
• Checklist is an observation

9
EXAMPLE Checklist for Prevention of Cross
Contamination Between Meats and Ready to Eat
Foods
? Uses separately labeled or color-coded cutting
boards ? Changes disposable gloves between
tasks ? Washes hands before and after, when
working with raw foods ? Cleans and sanitizes
food contact surfaces that touch contaminated
food before they come in contact with cooked or
ready to eat food ? Stores raw or contaminated
food below cooked or ready to eat foods to
prevent them from touching or dripping fluids
10
EXAMPLE Checklist for Clean Healthy Homes

• Of the 10 program participants, I have observed
  the following making changes
• ___ Number who now dust at least weekly.
• ___ Number who vacuum at least weekly.
• ___ Number who dispose of trash properly.
• ___ Number who change air filters monthly.

11
Examine Records

• Benchmarks
• But, records are often not available
• So, make your own
• Record/diary/checklist

12
Focus Group Interviews

• 5-7 people
• Ask them what happened as a result of the program
  
• Ask follow-up questions
• Record the answers
• Data can be hard to analyze, but its rich

13
Surveys

• Match question to scale
• Dont have bunches of scales and different
  questions
• As simple as possible
• Stick to what you have to know

14
Response Scales

• Another name is answer categories
• Match the responses to what you are measuring
  (outcomes)
• Measuring behavior or practice change
• Measuring attitudes or opinions
• Measuring skill improvements

15
Measuring Behavior or Practice Change

• 0Never
• 1-2 Once or Twice
• 3-5 3-5 times
• 6 6 or More Times

16
Aspirations

• Do you expect to
• As a result of this program
• Definitely No
• Probably No
• Probably Yes
• Definitely Yes

17
Attitudes/Opinions

• Strongly Oppose
• Oppose
• Neutral
• Favorable
• Strongly Favorable

18
How do you feel about a ban on smoking in all
restaurants?

• 50 __ 1. Strongly Oppose
• 0 __ 2. Oppose
• 0 __ 3. Neutral
• 0 __ 4. Favorable
• 50 __ 5. Extremely Favorable
• Mean 3.0 Neutral

19
How do you feel about a ban on smoking in all
restaurants?

• 30 ___ 1. Strongly Oppose
• 20 ___ 2. Oppose
• 0 ___ 3. Neutral
• 20 ___ 4. Favorable
• 30 ___ 5. Extremely Favorable
• Mean 3.0 Neutral

20
How do we report means?

• Five part knowledge scale
• Pre-test mean of 3.5 Post-test of 4.5
• 4.5 minus 3.5 1
• 5 divided by 1 20
• Report that there was a 20 gain in knowledge

21
Pre-Test and Post-Test

• No need to say pre-test or post-test to your
  participants
• Its time to begin. First, you will complete a
  very short exercise that will help you prepare
  for our program.
• The last part of our workshop will allow you to
  review what we did today.

22
Post-Then-Pre Common Example
Please think about your healthy indoor air
practices now and before this program. Please
check which, if any practices you do now (left)
and then check which, if any, practices you were
doing before this program (right).
NOW BEFORE THIS PROGRAM ____ 1. I use a
plastic mattress cover. ____ ____ 2. I dust
weekly. ____ ____ 3. I change air filters
monthly. ____ ____ 4. I open a bathroom window
to reduce moisture. ____ ____ 5. I vacuum
weekly. ____
23
How important is indoor air quality to you?

• Some people feel that indoor air quality is no
  big deal. Others are concerned. What is true for
  you? Please place a B on one step to show how
  important you thought indoor air quality was
  before our series. Please place an N on the
  step that shows how important you think indoor
  air quality is now.

Very Important
Not Important
24
Tally Results

• N 88
• Four step increase 50 people
• Three step increase 18 people
• Two step increase 10 people
• One step increase 9 people
• Same or Decrease 1 person

25
Changing Attitudes for Better Health

• UT Extension participants reported that they
  changed attitudes about the importance of indoor
  air quality for better health! 88 participants
  completed an attitude scale that described how
  important they felt indoor air quality was for
  better health. The scale measured their feeling
  both before and after the program, and the scale
  ranged from no change to a four level change.
  After the four-month program, 98 reported that
  they now view air quality as more important for
  their total health.
• The majority of participants (50 people 56)
  increased four levels on the air quality attitude
  scale.

26
Using Results of Open-Ended Questions

• Turn outcome into a question
• Obtain answers
• Later, write down answer categories
• Now list the participants answers
• Put answers in categories

27
Turn your outcome

• Participants will
• Understand rule 1018,
• Understand rule 406, and
• Understand health effects of lead exposure.

28
into a Question

• Did you learn anything new about lead poisoning
  prevention?
• Yes, No, Dont Know
• If yes, what did you learn?

29
Write down answer categories

• The Outcomes
• Understand rule 1018,
• Understand rule 406, and
• Understand health effects of lead exposure

30
List the participants answers

• Rules for realtors (406)
• No answer
• Lead exposure slows learning ability (health
  effects)
• How lead exposure affects brain development
  (health effects)

31
Figure Percentages

• Health Effects 20 of 50 40
• Rule 1018 10 of 50 20
• Rule 406 10 of 50 20
• No answer 5 of 50 10
• All other 5 of 50 10

32
Measuring Barriers to Change Example A

• Not interested in ___________________
• Program did not motivate me to get started
• Have not had enough time to get started
• No money to get started
• Found the directions totally confusing
• Dont do things like that
• Dont believe what I learned
• Other reason (please list___________)

33
Measuring Barriers to Change Example B

• In your opinion, why do more people not take
  their used oil and antifreeze to a service
  station or recycling center?
• What could be done here in _______
• County to make it easier for people to take
  their used oil and antifreeze to a service
  station or recycling center?

34
Assessment

• Considerations
• True/False
• Multiple Choice
• Short Answer, etc.

35
True/False

• Fast and easy
• Appropriate if objectives are identify, select,
  or recognize
• 50 is a weakness
• All, always, never, none and only - tend to be in
  false statements
• For clarity, avoid large, long time, regularly,
  some and usually

36
Short Answer/Completion

• Appropriate if objectives recall, remember,
  label, list, select, define
• Only one blank per question
• If answer is a number, indicate the unit of
  measure inches, gallons, , etc.
• Blanks near end of the sentence and same length
• Answers should be key terms

37
Multiple Choice

• Objectives recognize, distinguish between,
  select, estimate, infer, predict, relate,
  categorize
• Less likely to obtain the correct answer by
  guessing
• Reading level and reading speed

38
Typical MC Questions

• Which would be another name for
• Which is an example of
• What would happen if
• What is the best (worst, most important, least
  important)
• Which of the following would have the greatest
• Which of the following best describes

39
Format

• 1. What is the best-selling potted plant in
  Tennessee every December?
• A. Amaryllis
• B. Chrysanthemum
• C. Kalanchoe
• D. Poinsettia

40
Per Question

• True/False 15-30 seconds
• Multiple Choice (brief)
• 30-60 seconds
• Multiple Choice 60-90 seconds
• Short Answer 30-60 seconds
• Matching five items to 6 responses 2-4 minutes

41
Evaluation Tool

• Tennessee 4-H Life Skills Evaluation System
  (LSES)
• Go to
• http//www.utextension.utk.edu/4H/lifeSkills/eval/
  demo/

42
LSES is a database

• Website
• Agents select from valid, reliable surveys
• Enter results online
• Instant, aggregated data
• Measure life skills gained by youth through 4-H
  programs

43
(No Transcript)
44
(No Transcript)
45
Create/Print Survey
Print This Survey
46
Enter Data
15 10 25 60 50
47
View Survey Results
10 10 10 10 60
5 5 25 25 40
10 20 20 10 40
0 0 0 0 100
0 0 10 10 80
Mean for the instrument 4.2
48
Thank You!
49
References

• Barkman, S.J. (2002). A field guide to designing
  quantitative instruments to measure program
  impact. West Lafayette, IN Purdue Extension.
• Bennett, C. Rockwell. K. (1995). Targeting
  outcomes of programs (TOP) An integrated
  approach to planning and evaluation. Retrieved
  August 29, 2003, from University of Nebraska TOP
  Web site http//citnews.unl.edu/TOP/english/index
  .html.
• Boser, J. (2004). Curriculum, research and
  evaluation 560 Student assessment. Unpublished
  course materials. University of Tennessee.
• Diem, K.G. (2002, Dec.) Using research methods to
  evaluate your extension program. Journal of
  Extension, 40(6). Retrieved from
  http///www.joe.org/joe/2003december/a1.shtml.
• Diem, K.G. (2003, Feb.) Program development in a
  political world its all about impact. Journal
  of Extension, 41(1). Retrieved from
  http//www.joe.org/joe/2003february/a6.shtml.

50
References

• Ferrer, M. Jacob, S., Ferraru, T.M. (2001, Oct.)
  Two (or more) heads are better than one An
  application of group process to developing
  extension evaluation tools. Journal of Extension,
  39(5). Retrieved from http//www.joe.org/joe/2001
  october/a2.html
• Kiernan, N.E. (2001). Measuring barriers to
  change Tipsheet 27. University Park, PA Penn
  satte Cooperative Extension. Available
  http//www.extension.psu.edu/evaluation/tipsheets/
  measuring barrierstochange.pdf. Kiernan, N.E.
  (2001). How many answer categories are correct?
  Tipsheet 38. University Park, PA Penn State
  Cooperative Extension. Available at
  http//www.extension.psu.edu/evaluation/tipsheets/
  howmanyanswercategoriesarecorrect.pdf.
• Kiernan, N.E. (2002). Calculating and displaying
  findings from a stepscale Tipsheet 47
  University Park, PA Penn State Cooperative
  Extension. Available at http//www.extension.psu.
  edu/evaluation/tipsheets/calculatinganddisplayingf
  indingsfromstepscale.pdf

51
References

• Kiernan, N.E. (2002). How to use evaluation data
  to write different types of impact statements
  Tipsheet 66. University Park, PA Penn State
  Cooperative Extension. Available at
  http//www.extension.psu.edu/evaluation/tipsheets/
  howtouseevaluationdatatowritedifferenttypesofimpac
  tstatements.pdf
• Taylor-Powell, E. (2002). Program development in
  UW-Extension. Retrieved August 28, 2003, from
  University of Wisconsin-Extension-Cooperative
  Extension, Program Development and Evaluation
  Unit Web site http//www1.uwex.edu/ces/pubs/pdf
  /G3658_1.PDF
• UW-Extension. (n.d.) Program development in
  UW-Extension Getting started in Extension
  programming. University of Wisconsin Distance
  Education/Digital Media Unit.