Survey Design and Analysis: Survey Basics

1
Survey Design and AnalysisSurvey Basics

• Division of MH/DD/SAS
• Quality Management Team

2
What is a Survey?

• A survey is a method of gathering information
  from a sample of individuals.
• Individuals can include consumers, family
  members, employees, community members, or
  potential referral sources.
• This sample is usually just a fraction of the
  population being studied.

3
Why conduct a survey?

• In behavioral healthcare services, surveys are
  most often conducted to determine levels of
  consumer satisfaction with services or employee
  satisfaction levels.
• Surveys can be conducted to gather data on how to
  determine unmet needs of consumers or consumer
  outcomes.
• Surveys can also be conducted to determine how
  best to expand services or collaborate with other
  community service providers or stakeholders.

4
Survey sample

• In a survey, the sample is not selected
  haphazardly or only from those who volunteer to
  participate.
• Survey participants are scientifically chosen so
  that results can be reliably projected form the
  sample to the larger population.

5
How large must the sample size be?

• Key Terms
• Confidence Interval
• The range around a measurement that conveys how
  precise the measurement is.
• Confidence Level
• The confidence level tells you how sure you can
  be. It is expressed as a percentage and
  represents how often the true percentage of the
  population who would pick an answer lies within
  the confidence interval. Most researchers use the
  95 confidence level.

6
How large must the sample size be?

• Key Terms
• Population Size
• How many people are there in the group your
  sample represents.
• Percentage/Prevalence
• The proportion of individuals in a population
  having a disease. Prevalence is a statistical
  concept referring to the number of cases of a
  disease that are present in a particular
  population at a given time.

7
How large must the sample size be?

• First you must determine the sampling frame,
  that is, all members of the population being
  studied.
• All consumers currently enrolled in services
• All consumers who received at least one service
  during the past 12 months.
• All human service providers in an LME catchment
  area.

8
How large must the sample size be?

• Internet-based Sample Size Calculators can assist
  you in determining the appropriate sample size
• http//www.surveysystem.com/sscalc.htmone
• http//www.raosoft.com/samplesize.html
• However, your sample selection MUST be random

8
11/27/2013
9
Common survey methods

• Mail surveys
• Telephone surveys
• In-person interviews
• Internet/web-based surveys
• Focus Groups

10
Mail Survey

• Pros
• Easily reach a large geographic area, providing
  for potentially diverse array of answers.
• Most individuals are familiar with mail-based
  surveys, increasing the ease of completion.
• Mail formats are generally good for probing
  sensitive issues
• Cons
• No way to verify if the individual does not
  understand the question.
• Respondents need to be motivated in order to
  respond to the survey and mail it back in, a
  factor which often results in a low completion
  rate and response bias

11
More about Mail Surveys

• Cost Effective
• Allows for Geographical Stratification
• Increased Honesty with Responses
• Increased Need to Attend to Wording of Questions
• Attend to Design and Format of Survey (looks
  matter)
• Common Pitfalls
• Using White Pages for sample (does not allow for
  random selection)
• Booklets vs. Photocopies (presentation matters)
• No Follow-up Mailing (one mailing is not
  sufficient)

12
In-person Interviews

• Pros
• In-person interviews have a high completion rate
  and provide an opportunity for clarification.
• Individuals conducting the interview can probe
  the subject for answers, but this tactic can
  introduce bias.
• Cons
• While in-person interviews are often very
  thorough, they are also very expensive. Thus,
  many organizations will forgo in-person
  interviews in lieu of mail or phone based
  surveys.
• Social desirability refers to the process by
  which subject individuals modify their responses
  to survey questions based upon their perception
  of the interviewer's own opinion.

13
Phone Surveys

• Pros
• Phone surveys are one of the most commonly used
  methods.
• Phone surveys allow for clarification of complex
  or unclear questions, making them good for
  sensitive issues.
• Phone surveys generally target individuals during
  evening hours, the time when a majority
  individuals will be home and awake.
• Completed quickly with a low probability of
  interviewer bias because the individual cannot
  judge the interviewers appearance.

14
Phone Surveys

• Cons
• Individuals can ignore the interviewer by
  refusing to pick up the call, checking Caller ID,
  or immediately hanging up.
• Cell phone numbers are generally not available
  for interviewers to include in their random
  sampling which could leave much of the younger
  population unavailable for study (know your
  audience).
• Some behavioral health populations may not have
  access to a telephone.

15
Internet/Web-based

• Pros
• Internet surveys are easy and numerous.
• With a possible worldwide population available
  and programmable elements, the internet survey
  allows for customization across a broad spectrum.
  
• Databases can be automatically converted
  digitally, avoiding time consuming reentry.

16
Internet/Web-based

• Cons
• Internet surveys require technical skill on
  behalf of the responder and the survey
  constructor.
• Depending on the method used to distribute the
  survey (placement on website, mass email, etc.),
  internet surveys may not offer a true
  representation of general populations.
• Some intended respondents may not have access to
  computers or the internet. This might present
  problems with specific behavioral health
  sub-populations.
• Need to have an understanding of the reading
  comprehension of intended respondents.

17
Summary

• Telephone interviews are an efficient method of
  collecting some types of data and are being
  increasingly used. They lend themselves
  particularly well to situations where timeliness
  is a factor and the length of the survey is
  limited.
• In-person interviews in a respondent's home or
  office are much more expensive than mail or
  telephone surveys. They may be necessary,
  however, especially when complex information is
  to be collected.
• Some surveys combine various methods. For
  instance, a survey worker may use the telephone
  to "screen" or locate eligible respondents (e.g.,
  to locate older individuals eligible for
  Medicare) and then make appointments for an
  in-person interview