Cornerstones of a Quality Survey - PowerPoint PPT Presentation

1 / 34
About This Presentation
Title:

Cornerstones of a Quality Survey

Description:

Sample surveying is the art and science of coming close! ... Temptations to cut cost be reducing number of interviews -sampling errors. 16. Drop-off Surveys ... – PowerPoint PPT presentation

Number of Views:24
Avg rating:3.0/5.0
Slides: 35
Provided by: dallas76
Category:

less

Transcript and Presenter's Notes

Title: Cornerstones of a Quality Survey


1
Cornerstones of a Quality Survey
  • Sample surveying is the art and science of coming
    close!
  • Sample surveys yield accurate results when
    researchers succeed in avoiding four kinds of
    error
  • coverage error
  • sampling error
  • measurement error
  • non-response error

2
Coverage Error
  • Every member of the population that the
    researcher is trying to describe would have an
    equal chance of being selected for the sample.
    Coverage error is thus avoided
  • sampling frame survey population
  • ( list of respondents from (subset of the
    frame)
  • which sample is drawn)

Avoid Coverage Error
Consider who might have been excluded from the
survey and how they might differ with respect to
characteristics important in the study
3
Sampling Error
  • Enough people would be sampled randomly to
    achieve the needed level of precision. Thus,
    sampling error is avoided.
  • Can never be totally avoided unless you do a
    census. Remember only a subset of the population
    is used.
  • Control by increasing sample size

Avoid Sampling Error
Control error by selecting proper sample size
how much error can you tolerate? . 5 or do
you require more precision?.
4
Measurement Error
  • Clear, unambiguous questions would be asked so
    that respondents are both capable of and
    motivated to answer correctly.
  • Thus, measurement error is avoided.
  • Respondents answers to given question is
  • inaccurate
  • imprecise
  • cannot be compared usefully to other respondent
    answers

5
Measurement Error
  • Method effect - mail, telephone survey,
    face-to-face survey may all yield different
    results.
  • Questionnaire - words may confuse respondents,
    structure of question may be poor, answers may
    not be mutually exclusive.

6
Measurement Error
  • The interviewer- lead respondents, bias answers,
    reinterpretation of questions to respondents.
  • The respondent- May interpret questions
    differently according to their life experiences
    or perceptions. Cant get the correct answer

Avoid Measurement Error
Avoid biased or vague questions or other areas
where measurement error might creep into the
results of the study.
7
Non-response Error
  • Everyone in the sample responds to the survey, or
    non-respondents are similar to respondents on
    characteristics of interest in the study.
  • Thus, non-response error is avoided.
  • Under-counts result from non-respondents may lead
    to inaccurate interpretation of survey results.

Avoid Non-response Error
Design the questionnaire and implement the
surveys to avoid non-response error. Response
under 60-70 is a red flag generally!
8
Improve Survey Quality
  • Control for
  • coverage error
  • sampling error
  • measurement error
  • non-response error

9
Decide what information you need.
  • What problems are you trying to solve?
  • What new information do you need to solve it?
  • What is the purpose of the survey?
  • Identify the problem
  • Clarify vagueness
  • Eliminate bias
  • Focus clearly on what you need to know not
    nice to know

10
Think in Terms of Results
  • What do you want to get from the survey?
  • Conceptualize this by preparing hypothetical
  • tables of results. This
  • forces us to separate need to know from nice
    to know
  • Focus Groups or directed discussion groups.
  • Help us to develop clarity in knowing which
    questions to ask in our survey. Stimulates
    thinking on a particular subject.

11
Choosing a Survey Method
  • Mail Surveys
  • Telephone Surveys
  • Face-to-Face Interviews
  • Drop-off-Survey

12
Choosing a Method
  • Evaluate the method based on
  • Specific Study topic and population
  • Budget- how much can you spend on the project in
    time, effort and financial resources
  • Staff - volunteers, paid, experienced
    non-experienced
  • Time constraints - How soon do you need the
    results of the survey? What results must be
    produced?

13
Mailed Surveys
  • Best suited for -
  • When reliable addresses are available.
  • Respondents likely to respond accurately and
    completely in writing.
  • When immediate turnaround is not required.
  • When money, qualified staff and professional help
    are scarce.
  • Greatest weaknesses -
  • Non-coverage error, incomplete lists.
  • Some are less likely to respond than others,
    response error.
  • Respondent ability to read and follow directions.
  • Little control over who fills the survey out.
  • Skipped items - item non-response.

14
Telephone Surveys
  • Best suited for-
  • Populations likely to have telephones.
  • Questions are straightforward.
  • Experienced help is available to conduct the
    interview.
  • Quick turn around time is important
  • Greatest weaknesses -
  • Not all people have telephones.
  • Directories are incomplete and inaccurate,
    multiple listings. ( random digit dialing and
    add-a-digit dialing)
  • Sensitivity to measurement error. Leading
    question interviewers.

15
Face-to-Face Interviews
  • Best suited for -
  • Surveying a population for whom there is no list.
  • Collecting responses from those who are unlikely
    to respond via mail or phone.
  • Complex questionnaires
  • well funded projects - cost of professional
    interviewers
  • Greatest weaknesses -
  • Expensive, and time consuming.
  • Callbacks add greatly to cost.
  • Requires well trained interviewers.
  • Temptations to cut cost be reducing number of
    interviews -sampling errors.

16
Drop-off Surveys
  • Best suited for -
  • Small community surveys in which are in a small
    geographic area.
  • Short and simple questionnaire.
  • Projects with small staff but large sample size.
  • Greatest weaknesses -
  • Time to get questionnaires delivered and picked
    up.
  • In ability to deliver questionnaire to the right
    individual.
  • Delivery to a mail box which is illegal!

17
Selecting a Survey Sample
  • A sample is a set of respondents selected from a
    larger population for the purpose of the survey.
  • How much sampling error is acceptable? - Function
    of how large the sample needs to be.
  • Three steps in sampling
  • define the survey population
  • obtain an adequate population list - list frame
  • select the sample

18
How Large Should a Sample Be?Sample size depends
on
  • How much sampling error can be tolerated.
  • Smaller samples cheaper to collect info.
  • Population size.
  • Small populations gain little from sampling
    especially if need for precision is great.
  • How varied the population is with respect to the
    characteristics of interest
  • The smallest subgroups within the sample for
    which estimates are needed.

19
Sampling Rules of Thumb
  • For large populations 100,000 - 1,000,000
  • we need approximately the same sample size. Only
    when we compare groups of less than several
    thousand does sample size make a difference.
  • The more diverse or variable populations require
    larger sample sizes. Take the conservative
    approach with respect to sample size

20
Sampling Rules of Thumb
  • Sample size is influenced by not how large the
    overall population is but sample size is based on
    how big the subgroup being studied is. Sampling
    error depends on the size of the subgroup.
  • Sampling error rates - Confidence Levels
  • 3
  • 5
  • 10

See Sampling Table
21
Finding Good Lists
  • Easy for small specific populations
  • More difficult for general populations of state
    or national level.
  • Get official sponsorship to improve list
    availability.
  • Concept that every member of population has an
    equal and independent chance of selection

22
Uncomplicated Sample Designs
  • Simple random sampling (SRS)
  • a lottery (drawing out of a hat)
  • random number table - how to use a table
  • computer generated lists of random numbers
  • Probability and Non-probability Sampling
  • Non-probability- purposive sampling
  • Results generalized from group likely not
    probable
  • Focus groups good example

23
Sampling Considerations
  • Face -to- Face Surveys
  • small area cluster sampling Map based block map
    samples
  • advantages in reducing travel time to collect
    sample
  • larger area cluster sampling- two-stage approach
    400 blocks 40 selected then sample within this
    frame.
  • Telephone Surveys
  • avoid coverage errors
  • Random digit dialing (RDD)
  • 3 digit prefix plus random computer generated 4
    digit extensions
  • Add-a-digit sampling
  • random number selected e.g. 5 added to random
    selected numbers
  • 883-0527. 5 .883-0532

24
Issues to Consider When Writing Survey Questions
Operationalizing Your Questions
  • avoid emotional and biased questions
  • how specific the question should be
  • whether the questions will produce credible
    information
  • whether respondents are able to answer the
    questions
  • whether respondents will be willing to provide
    the information.

25
What Kind of Information are You Looking For?
Clarify What You Want to Know
  • Four Types of Questions
  • What people do or what they are
  • Behavior or Attributes
  • What people say they want or what they think is
    true.
  • Attitudes or Beliefs

26
Open- Ended Questions
  • Open-ended questions do not provide choices to
    select from. Respondents formulate the answer
  • Drawbacks
  • demanding for respondents
  • difficult to analyze the data- time to prepare
    for computer entry or hand analysis
  • produce many different responses
  • rarely provide accurate consistent measurement
    when comparing whole group

27
Open-Ended Questions
  • Advantages
  • You have little prior knowledge about an area
  • Want to give respondents opportunity to state
    strong opinions, vent frustrations
  • Followed by a close ended question - respondents
    explain response
  • When respondents are asked to estimate a routine
    behavior and they are likely to know an exact
    number
  • When a precise piece of information is needed and
    cannot easily be listed in a closed ended
    question.

28
Close-ended with Ordered Response
  • Each question choice represents a graduation of a
    single concept.
  • 1. Under 25 years
  • 2. 26-35 years
  • 3. 36-45 years
  • 4. 46-55 years
  • 5. 56-65 years
  • 6. over 65 years
  • Less demanding for respondent, quite specific and
    easier to code and analyze.

29
Close-ended with Unordered Response Choices
  • Responses do not fall on a continuum
  • Unordered response choices generally more
    difficult to answer
  • Often used to rank items -
  • rank these items 1-6.
  • these types of questions are difficult with
    telephone surveys. We loose track and cant
    remember.

30
Partially Close-ended Questions
  • Compromise between open-ended and close-ended
    structures. Respondents have the opportunity to
    add their own response.
  • Other ( Please Specify)__________________
  • this format rarely yields additional responses
  • Gives respondent choice to not be boxed in with
    their response. May generate new information

31
Which Question Structure Shall I Use?
  • Question structure affects how people respond.
    Category Effect
  • Mail survey -respondents more likely to choose
    first categories - telephone and face-to-face the
    last categories
  • When list of choices gets too long
  • Abstract ideas about which people havent given
    much thought. Use a series of leading questions-
    Cognitive Design
  • Use a series of questions rather than one complex
    question - Scaling Technique
  • Be aware of how category effect can increase
    measurement error.

32
There are NO Silver Bullets When Writing Survey
Questions
  • Be Specific
  • Use simple Words
  • Dont be Vague
  • Keep it Short
  • Dont talk down to respondents
  • Dont Be too specific

Conflicting Rules
33
Avoid these Question Traps
  • Vaguely worded questions and responses
  • Abbreviations or jargon that respondents may not
    understand
  • Too much precision
  • Bias from slanted introduction
  • Bias from unequal comparison
  • Bias from unbalanced response choices
  • Bias from tone of question
  • Objections to providing income information

34
Avoid these Question Traps - continued
  • Objectionable statements
  • Questions that are too difficult for respondents
  • Double-barreled questions
  • Answers that are not mutually exclusive
  • Too much knowledge is assumed on the part of
    respondent
  • Inaccurate statements
  • In appropriate time references
  • Responses that cant be compared with existing
    information
  • Cryptic questions
Write a Comment
User Comments (0)
About PowerShow.com