Collecting Evaluation Data

1
Collecting Evaluation Data

• AEE 577
• Class X

1, Yes 2. No
Respondent
2
Objectives

• Upon completion of this lesson, students should
  be able to
• Describe advantages of sampling
• Describe common methods of sampling
• Determine the sample size based on the size of
  the target audience
• Draw a random sample
• Draw a stratified random sample
• Describe strategies for increasing response rate
• Describe nonresponse error controlling techniques.

3
Sampling
4
Sampling

• Most follow-up evaluation surveys involve
  selecting a sample because of the cost and time
  involved in surveying the entire population.

5
Advantages of Sampling

• Sampling will save
• Time
• Money

6
Activity 1 Determining Sample Size
If this sack contains 180 MMs of 3 different
colors, how many would you need to draw out to
get an accurate estimate of the percentage of
each color in the sack?
7
How Big Does the Sample Need to Be?

• Researchers and statisticians have developed
  formulas and tables that show how big the sample
  has to be.
• Generally, two things are needed in order to use
  these tools
• How big is the population?
• How much chance error are you willing to accept
  (confidence level and confidence interval)

8
Common Sample Size Experts

• Cochrans Sample Formula
• Cochran, W. G. (1977). Sampling techniques (3rd
  ed.). New York Wiley
• Krejcie Morgan
• Krejcie, R.V. Morgan, D.W. (1970). Determining
  sample size for research activities. Educational
  Psychological Measurement, 30, 607-610.

This is used extensively in agricultural and
extension education studies
9
Krejcie Morgan

• The formula for determining sample size developed
  by Krejcie and Morgan is shown below, but they
  have made several trial calculations and have
  developed a table that is simple to use.

See Next Slide
10
Krejcie Morgan Table
This is the Krejcie and Morgan table. Most people
use the first column. Go to the next highest
number if your exact population is not shown.
11
Cochran

• With the Cochran formula, you have to plug in
  data and manually calculate an answer. It is
  somewhat complicated.

12
More Krejcie and Morgan

• For exact sample sizes for smaller populations, a
  table found at this web site gives those numbers.
• http//www.sageperformance.com/drjeffallen/DrA/Tea
  ching/5480/samplesize.htm

13
An Easy Way to Determine Sample Size

• Go to http//www.macorr.com/ss_calculator.htm
• and enter your figures.

14
The Mechanics of Selecting a Sample

• Put everyones name on a piece of paper and draw
  names out of a hat. (not very efficient use of
  time for large groups)

15
The Mechanics of Selecting a Sample

• Use a table of random numbers
• Number all the people in the population, then use
  a table of random numbers (found in statistics
  books or on the web) to identify which
  individuals to select.

16
Selecting a Sample

• Go to http//www.randomizer.org/form.htm and have
  numbers automatically generated for you.
• Or you could do this in Excel
• Will produce a random whole number between 1 and
  500.

17
Other Views about Sample Size

• According to Gay Diehl, (1992), generally the
  number of respondents acceptable for an
  evaluation depends upon the type of study
  involved - descriptive, correlational or
  experimental.

18
Gay and Diehl (1992)

• For descriptive research the sample should be 10
  of population. But if the population is small
  then 20 may be required. (Would a 20 sample of
  the MMs give you a representative sample?)

19
Gay and Diehl (1992)

• In correlational studies at least 30 subjects are
  required to establish a relationship.
• For experimental studies, 30 subjects per group
  is often cited as the minimum.

20
Types of Samples

• Probability Sampling
• Regarded as the best most scientific
• Everyone in the population has an equal chance of
  being selected
• Non-Probability Sampling
• Non-scientific
• Sample may not be (generally isnt)representative
  of the general population

21
Probability Sampling

• Simple Random Sample
• Each individual in the population has an equal
  chance of being selected.
• An example Put everyone's names in a hat and
  then draw them out.

22
Activity 2 Simple Random Sample
Draw out a simple random sample of the MMs
according to the number you said in Activity 1
23
Activity 3 Simple Random Sample
Put the MMs back in the sack and then draw out a
simple random sample of the MMs according to the
Krejcie Morgan table.
24
Probability Sampling

• Stratified Random Sample
• Used to ensure that sub-groups within a
  population are represented proportionally in the
  sample.

25
Stratified Random Sample
You select a percent of the sample from each
sub-population in the same proportion as they are
in the population.
134 Male Ag Teachers
57 Female Ag Teachers
Stratified Sample of Ag Teachers in NC (70 male,
30 female)
266 Male Ag Teachers
114 Female Ag Teachers
Population of Ag Teachers in NC (70 male, 30
female)
26
Activity 4 Stratified Random Sample
Divide your MMs into 3 groups according to color
and count the numbers in each group. This is your
population. Determine how many would need to be
drawn from each group for a stratified random
sample.
27
Activity 4 Stratified Random Sample
If you dont have time to count MMs use this
table and determine your stratified sample.
28
Activity 4 Stratified Random Sample
The Answer Key Number in each group to include
in a stratified random sample.
29
Probability Sampling

• Cluster Sampling
• Random selection of groups that already exist.
• Example To conduct a study with Horticulture I
  Ag students, you would randomly select schools,
  then randomly select Hort I classes from within
  the schools

30
Cluster or Multi-Stage Sampling
Regions
Region 5
Region 2
Counties
Cumberland
Davie
Wilkes
High Schools
Sampson
Hort I Classes
High School 4
High School 3
High School 1
High School 2
3rd Period Hort 1
4th PeriodHort I
3rd Period Hort I
1st Period Hort I
31
Probability Sampling

• Systematic Random Sample
• This is random sampling with a system!  From the
  sampling frame, a starting point is chosen at
  random, and thereafter at regular intervals.  

32
Systematic Random Sampling Examples

• The sample is drawn from a numbered list of
  people. A person is randomly picked near the top
  of the list, then every Nth name is selected
  after that (Nth could be 3rd, 7th, 10th or
  whatever number is needed to get the correct
  sample size).
• You could sample houses on a street, hours of the
  day, telephone numbers in a phone book, customers
  in a line, etc.

33
Systematic Random Sample (every 3rd person
selected)

• Bob Adams
• Billy Benham
• Sue Conners
• Ward Dunlap
• Teresa Elgin
• Bob Franks
• Cindy Gomez
• Dan Headley
• Aaron Jackson

• Sue Kimmons
• Todd Larson
• Barb Morris
• Helen Newcomb
• Inez Oppenheimer
• Tad Porter
• Linda Rush
• Robert Sims
• Tina Thompson

34
Non-Probability Sampling

• Convenience (also called accidental sample)
• The evaluator selects whomever is convenient
• Example A evaluator at the mall selects the
  first five people who walk by to get their
  opinion of a product.

35
Non-Probability Sampling

• Purposive (or judgmental sample)
• Individuals are selected because of their
  expertise, specialized knowledge, or
  characteristics.
• Example To learn more about emerging trends or
  issues in the field, you might want to survey the
  professional organization leaders.

There are times when this might be desired. For
example, studying the top teachers or agents in
the state may provided more useful information
than studying random folks.
36
Non-Probability Sampling

• Snowball Sampling (also know as chain or network
  sampling)
• A small group is initially identified . After
  data are collected from them, they are asked to
  identify others who might have specialized
  knowledge regarding the topic those thus
  identified recommend others.

37
Response Rate

• Number of surveys returned/Number of surveys
  mailed
• Getting a good response rate is a challenge

38
Factors Contributing to Low Response

• Length of the survey The longer the survey the
  lower response rate
• Time taken to complete the survey The longer the
  time the lower the response rate
• Open-ended questions The greater the number of
  open-ended questions in the survey the lower the
  response rate
• Clarity of the survey The lower the clarity of
  survey the lower the response rate
• Sensitive information The greater the number of
  potentially sensitive questions such as
  demographic info. the lower the response rate.

39
Strategies to Enhance Response Rate

• Use closed-ended questions
• Limit open-ended questions to the minimum
• Use easy to answer format
• Keep it short and clear
• Include a personalized cover letter and explain
  the purpose of the survey and sign it
• If doing a mail survey, include a stamped and
  addressed return envelope.
• Choose the less busy time or season for the
  respondent
• If you have adequate resources, you may offer a
  drawing for prizes for the respondents

40
Nonresponse Error

• If the characteristics of your respondents are
  different from those of nonrespondents, then
  nonresponse error can take place.
• Therefore, when you have nonrespondents you need
  to verify whether they are different from the
  respondents to make valid recommendations for the
  total population.

41
Controlling Nonresponse Error

• There are two practical approaches
• Comparing early to late respondents
• Comparing respondents to a random sample of non
  respondents
• If there is no significant difference between the
  respondents and nonrespondents, then you can
  generalize results for the total population.
• If there is any difference, then you need to make
  adjustments before making any recommendations for
  the entire population.
• If you have 85 or above response rate, then you
  may not need to address nonresponse error.

42
Summary

• Objectives
• Sampling
• Advantages
• Common sampling techniques
• Response rate
• Nonresponse error