Survey Research

1
Survey Research
2
Survey Research - Definition

• Survey research is the method of gathering data
  from respondents thought to be representative of
  some population, using an instrument composed of
  closed structure or open-ended items (questions).
• The goal is to describe some aspect or
  characteristics (opinions, knowledge, attitudes,
  beliefs) of a population.

3
Survey Research General Comments

• A widely used form of data collection in the
  social sciences
• Allows for efficient collection of data over
  broad populations
• Can be implemented in person, by telephone, and
  over the Internet.
• Criticisms
• Artificially forces respondents to formulate
  opinions
• Tends to simplify complex issues
• Advantages
• Not as hard to do as some other types of research
• Appropriate method for answering questions in our
  field.

4
Two Major Types of Surveys

• There are two major types of surveys
• Cross-sectional
• Collects information from a sample of the
  population at a specific point in time (one and
  done)
• Information is collected at just one point in
  time, even though it could take days to gather
  all the data
• If you survey everyone in the population, that is
  known as a census

5
Two Major Types of Surveys

• Longitudinal
• Information is collected at different points in
  time in order to study change over time
• Graduate students typically dont do this type
  of research because they want to graduate
• There are three categories of longitudinal
  surveys (described on the next three slides)

6
Types Of Longitudinal Surveys

• Trend survey the same population every year,
  but the same individuals may not be selected
  (randomly select a group each year from the total
  population)
• If a researcher conducted an annual survey of job
  satisfaction of extension agents every year, he
  would draw a random sample out of this population
  every year.
• Because of retirements, new hires, etc. the
  population could change a little every year
• By comparing results from year to yearone can
  see trends that are developing.

Year 2 Group
7
Types Of Longitudinal Surveys

• Cohort study Samples are drawn from a
  population that remains the same. However, the
  people sampled may vary from year to year.
• If you wanted to study all the CALS graduates
  from 2009, you would drop a sample from that
  population.
• The next and subsequent years, you would
  continued to draw a sample from the class of 2009

8
Types Of Longitudinal Surveys

• Panel Study A sample is drawn from a
  population, and those some individuals are
  studied over time.

9
Correlational Research

• Most survey research is descriptive, however one
  can also use surveys to collect data that can be
  used in correlational research and ex post facto
  studies.

10
Steps Involved in Survey Research

• Clearly define the problem to be investigated.
• Formulate research questions or objectives
• Identify the population to be studied
• Select the sample
• Choose the mode of data collection
• Mail Surveys
• Electronic Surveys
• Direct Administration to a Group
• Telephone Surveys
• Personal Interviews

11
Mail Surveys

• Usually among the least expensive modes, are
  often best for sensitive items, and there is no
  interviewer bias. However, mail surveys are a
  poor choice for open-ended items or complex
  survey designs as they place a high burden on the
  respondent. Other liabilities include lack of
  control over respondent interaction with unknown
  others while taking the survey and while being
  faster than face-to-face administration, mail
  surveys are slower than phone or web surveys.
  Mail surveys are often used for very short
  instruments.

12
Direct Administration to a Group

• Is used when a researcher has access to all (or
  most) of the members of a particular group in one
  place. The instrument is administered to all
  members of the group at one time. The chief
  advantage is close to 100 response rate, low
  cost, and the researcher can explain the study
  and answer questions.

13
Personal Interview

• Is usually the most expensive but it can obtain
  the highest response rate because it maximizes
  engagement with the respondent and imposes the
  lowest burden on the respondent. For the same
  reasons, face-to-face interviews can support
  longer instruments. It is also the best data
  collection type for open-ended responses. In
  situations where the respondent is initially
  unknown, this may be the only feasible data
  collection type. Liabilities of this method
  include possibility of interviewer bias,
  respondent bias on sensitive items, slow data
  collection time, and cost. Face-to-face
  interviewing is often used when the research
  purpose requires in-depth exploration of
  opinions.

14
Electronic Surveys

• Are often the least expensive to administer yet
  can be fast in terms of data collection,
  particularly since implementation time may be
  low. Web surveys can be administered to very
  large populations, even internationally. Also,
  they support complex survey designs with
  elaborate branching and skip patterns depending
  on the response. The central drawback to web
  surveys is the near-impossibility of drawing
  random samples since web access is known to be
  biased by income, race, gender, and age. Also,
  web surveys impose almost as high a burden on the
  respondent as do mail surveys, and thus are not
  optimal for long instruments or open-ended
  questions. Web surveys are frequently a good
  choice when an unscientific but quick straw
  poll is needed on some subject.

15
Telephone Surveys

• Has the advantage of speed of data collection
  while supporting longer instruments than mail
  surveys and supporting open-ended responses,
  though not as well as face-to-face interviewing.
  There is also moderate control over interviewer
  bias since interviewers follow predetermined
  protocols, and there is high control over
  respondent interaction with others during the
  interview. Telephone interviewing can be a good
  moderate-cost compromise between mail and
  face-to-face survey research.

16
Advantages and Disadvantages of Survey
Data-Collection Methods
17
Survey Research Steps cont.

• Construct the survey instrument
• No matter which mode of data collection is
  employed, there must be an instrument for
  collecting the data.
• The validity of the instrument needs to be
  established (refer to the previous lesson)
• The reliability of the instrument must be a
  matter of concern (refer to the previous lesson).

18
Dillmans Total Design Method

• Donald Dillman, a professor at Washington State,
  has developed a process for conducting survey
  research that he says will yield a 80 response
  rate.

19
Tailored Design MethodElement 1
Respondent-Friendly Questionnaire

• easy and clear to understand
• relevant question order
• comprehensible, user-friendly layout design
• Also
• make the questions interesting
• consider shortening the questionnaire

20
Respondent Friendly Questionnaire

• Construct the questionnaire in booklet form
• 8½" x 14" legal paper
• Creates an 8½" x 7" booklet
• Folded length-wise will fit in a standard
  business stationary envelope
• 11" x 17 paper
• Creates an 8½" x 11" booklet
• Can consider two columns per page
• Put an appealing graphic and title on the front
  cover.
• Try to limit to 12 total pages or less
• Include instructions in the booklet

21
Survey Design Considerations

• Survey Layout
• For mail and web surveys, the survey layout
  itself needs to be attractive, easy-to-use, and
  non-intimidating.
• Include plenty of white space (as opposed to
  dense single-spaced print)
• Have response areas be consistent and easy to
  check off (checkboxes are clearer than blank
  lines)
• Have response areas on the right since most
  respondents are right-handed
• Use one or at most two colors to set of sections
  and items, avoiding distracting color schemes.
• Avoid multiple fonts, italics (harder to read),
  animations, blinking font
• Print the survey on colored paper, or at least
  the cover

22
Choose the 1st Question Carefully

• No question is more critical than the first one.
• Will more likely determine whether survey is
  returned or trashed.
• Should be easy so that all respondents will need
  only a few seconds to read, comprehend, and
  respond to it.
• The first question should be clearly related to
  the announced purposes of the survey (not a
  background item, for instance).
• It should be closed-ended
• Interesting
• Applicable to everyone in the study

23
Order of Other Questions

• A questionnaire is like a conversation.
• One question should lead to the next like a
  person listening during a conversation.
  Constantly changing topics is confusing.
• Begin by order questions from most salient to
  least.
• Sensitive (objectionable) questions should go in
  the middle or near the end of the survey.
• Demographic questions typically are at the end
  does not require much thinking
• Generally questions are grouped according to
  their format.

24
Additional Thoughts

• It is very tempting to leave out 'no opinion,'
  'not sure,' and 'don't care' options in
  structured questions. Forcing a choice not only
  yields the direction of preference, but also
  requires the respondent to give a more considered
  response. However, forcing a choice does distort
  the responses of individuals who genuinely have
  no opinion, are genuinely unsure, or who
  genuinely do not care.

25
Survey Research Steps cont.

• Field test the instrument with a group similar to
  your target population to uncover any
  unanticipated problems or issues and to refine
  the instrument.
• Converse and Presser (1986) argue that two
  pretests are needed and they suggest having 25-75
  individuals in the pretest.
• The first pretest is for major revision, the 2nd
  is to polish
• In a field test you might have more items than
  really needed to see which items perform the
  best. Drop the weaker items from the final survey.

26
Field Testing continued

• Open ended questions could be utilized to
  determine possible response items for the final
  survey.
• Field tests reveal practical information -- Is
  there enough space to answer open-ended
  questions? Are instructions clear? Etc.

27
Survey Research Steps cont.

• Collect the Data
• The slides that follow immediately refer to
  mailed questionnaires
• Slides that follow after that will look at other
  data collection processes

28
Tailored Design MethodElement 2 Four Contacts
by First-Class Mail,with an Additional Special
Contact
It is desirable to provide a deadline for
returning the survey.

• 1. pre-notice letter
• sent to respondents a few days prior to the
  questionnaire
• 2. questionnaire mailing (see following slide)
• sent a few days to a week after the pre-notice
• 3. thank you/reminder postcard
• sent about one week after the questionnaire
• 4. replacement questionnaire
• sent to nonrespondents, 2-4 weeks after
  questionnaire mailing
• 5. final contact
• made a week after the replacement questionnaire
  by special delivery or telephone

29
Second Contact

• The initial questionnaire mailing. Packet to
  include
• Cover Letter
• Appropriately constructed, letterhead, signed
• Questionnaire
• Booklet, saddleback stapled, figure on front
  page, back page blank.
• Pre-addressed, stamped (real), return envelope
• Incentive

30
Third Contact

• A thank you postcard
• Sent a few days to a week after the
  questionnaire.
• The mailing expresses appreciation for
  responding, and indicates that if the completed
  questionnaire has not yet been mailed it is hoped
  that it will be returned soon.

31
Fourth Contacts

• A replacement questionnaire
• Sent to nonrespondents 2-4 weeks after the
  previous questionnaire.
• Indicates that the persons completed
  questionnaire has not yet been received and urges
  the recipient to respond.

32
Final Contact

• May be made by telephone a week or so after the
  fourth contact (if telephone numbers are
  available).
• This special contact has been shown to add an
  additional 13 response
• Priority Mail, UPS delivery etc. should also be
  considered

33
Tailored Design MethodElement 3 Return Elements
with Real First-Class Stamps

• The use of a real stamp on return envelopes
  represents
• a goodwill gesture
• something of value
• something the respondent is less likely to throw
  away
• quite possibly, an increase in response by
  several percentage points

34
Tailored Design MethodElement 4 Personalization
of Correspondence

• Personalize correspondence by using
• real stationery
• real names
• real signatures
• or, by sending replacement questionnaires with
  the statement, To the best of our knowledge you
  have not yet responded.
• Have a noted person co-sign the cover letter

35
Cover Letter

• The cover letter is virtually the only
  opportunity the researcher has for anticipating
  and countering respondent questions.

36
Cover Letter (what to include)

• Explain what the study is about
• Convince respondent study is useful
• His/her response is important to study
• Promise confidentiality
• Reemphasize social usefulness
• Pressed blue ball point signature

37
Tailored Design MethodElement 5 Token Prepaid
Financial Incentives

• A small token, financial incentive (e.g., 2) can
  significantly improve response rates.
• Promised incentives have had modest and, in some
  cases, no effect at all.

38
Other tips

• Avoid Subordinating Language
• Your ideas about how to solve the school
  problems are important vs. Would you please do
  me a favor?
• Make it Short and Easy
• Questionnaires that appear shorter and easy to
  fill out lessen increase response rates
• Such appearance can be reinforced by indicating
  in the cover letter that responding should only
  take a few minutes
• Appeal to people on the basis that something
  important will ultimately happened as a result of
  the survey.

39
Conducting a Mailed Survey

• Key Points
• Get as many responses as possible
• Control non-response error.

40
Internet or Web-based Surveys

• Becoming very popular
• Principles for designing paper based instruments
  also apply to electronic instruments
• CALS has a home grown electronic survey
  instrument.
• It may not be as jazzy as some of the commercial
  sites (surveymonkey, zoomerang, etc.) but it is
  free
• The Survey Builder tool can be found at
  http//ceres.cals.ncsu.edu/websitebuilder

41
Activity
http//ceres.cals.ncsu.edu/websitebuilder

• Assume you want to conduct a study of the job
  satisfaction of extension agents, agriculture
  teachers or some similar group.
• Prepare five questions to be asked (dont spend
  an inordinate amount of time on this task).
• Question 1 should be closed-ended (I would
  suggest a Likert type question.
• Question 2 should be open-ended
• Question 3 should be all that apply
• Question 4 should be multiple choice
• Question 5 is your choice
• Go to the computer lab, open up the CALS survey
  tool, and prepare your survey.

42
Mixed Mode or Bi-modal Survey

• Some recent work on combining methods.
• Start with web and use mailings for subsequent/
  final rounds.

43
Phone Surveys
44
Pros and Cons of Phone Surveys

• Pros
• Higher response rates
• Ease of implementation
• Possible to explore complex issues
• Can gather more information in less time than a
  mailed survey
• Greater depth of response
• Can gather information quickly
• Literacy is not an issue

• Cons
• Cost (but not as prohibitive as it once was)
• Some populations are not readily accessible
• Low income
• People in institutional settings
• Yuppies

45
Phone Survey Steps

• Develop and pretest the telephone survey
  questionnaire
• Obtain the phone numbers
• Typically for research in agricultural and
  extension education, we know who we are surveying
  and have the phone numbers
• Otherwise one can randomly select phone numbers
  from a phone book
• Some years ago Oklahoma State randomly called
  900 people to see if they knew anything about
  the Cooperative Extension Service
• Caution Not all folks are listed in the phone
  book, this could bias your results

46
Phone Survey steps continued

• Decide who in the household will provide the most
  meaningful input to the survey and ask for that
  person.
• Call each phone number. Politely explain the
  purpose of the survey and tell how much time the
  survey will take.
• Dont talk fast this is what sales people do
• Stress the importance
• This is a very important survey designed to
  directly impact the services and programs
  available to your children, and it should only
  take about ten minutes to complete. Is now a good
  time for you?
• Offer to call back at a more convenient time

47
Phone Survey steps continued

• Read each question in its entirety to the
  participant and record the response.
• Repeat the question if necessary but dont
  interject comments or explanations.
• Record the responses
• Keep the participant on track so the survey is
  completed as quickly as possible

48
Use a Call Sheet

• A call sheet is a log on which to record
  information
• Phone number
• Time called
• Outcome of call

49
Personal Interviews
50
Personal Interviews

• Start with an introduction to the research and
  its importance.
• Good morning (afternoon). My name is ___. Thank
  you for coming. This interview involves two
  parts. The first part is a survey, in which I
  will ask you about your experiences as a student
  at this university. The purpose is to get your
  perceptions of your experiences inside and
  outside of the classroom. There are no right or
  wrong or desirable or undesirable answers. I
  would like you to feel comfortable with saying
  what you really think and how you really feel.
  The second part is a short pencil-and-paper task,
  and I will give you specific instructions for
  completing that task once we have finished with
  the survey.

51
Another sample introduction

• My name is Margaret Moore. I am working on an
  approved research study at Virginia Tech
  regarding the connection between school-based
  learning and work-based learning. The study will
  involve ten school sites in southwest Virginia.
  Do you have any questions at this time?Thank
  you for your willingness to participate in this
  research project. Before we begin the interview,
  I would like to reassure you that this interview
  will be confidential and the tape and transcripts
  available only to me. Do you mind if I record the
  interview? ______ltif yesgt If there is anything
  you dont want me to record just let me know and
  I will turn off the recorder .Excerpts of this
  interview may be made part of the final research
  report, but under no circumstances will your name
  or identifying characteristics be included in
  this report. Is it all right for me to turn on
  the recorder now?

52
Personal Interviews

• Follow the interview protocol (questions)
• Q1. What is your major?
• Q2. What year did you graduate from high school?
• Q3. Did you participate in the Farm Bureau IFAL
  Program?
• Yes (Go the Q3a)
• No (Skip to Q4)
• Q3a. What were the most helpful aspects of that
  program?

53
Interview protocol continues

• Q4. Are you a member of any CALS student
  organizations on campus?
• No (Skip to Q5)
• Yes (Go to Q4a Q4b)
• Q4a. What are those organizations?
• Q4b. What are the most helpful aspects of
  that/those organization(s)?

54
Personal Interviews

• Record the responses
• Thank the person for their time and input at the
  end

55
Survey Research Steps cont.

• Analyze the data
• Prepare the research report

56
Evaluating Threats to Internal Validity in Survey
Research

• There are four main threats to internal validity
  in survey research
• Mortality
• In longitudinal research, subjects become lost.
• Location
• Interviews could be conducted in a place that
  could affect responses.
• Instrumentation
• Instrument is biased, poorly constructed, etc.
• Instrument decay
• Interviewer gets tired or rushed

57
Non-response Error

• Non-response is a serious problem in many
  surveys.
• Members of the sample cannot be reached
  (non-deliverables) or refuse to respond
  (non-respondents)

Researchers can not ignore non-response error. It
must be addressed
58
Proposed Procedures for Handling Nonresponse
Issues

• When 85 not achieved
• Comparison of early to late respondents
• Compare respondents to nonrespondents
• Contact a random sample of non-respondents by
  phone and record their information

59
Survey Research

• Require a great deal of research knowledge and
  sophistication.
• Looks easy but isnt
• Sampling
• Handling non-response error
• Design, construction, and testing of
  questionnaires and interview schedules get to be
  major undertakings.

60
References

• Converse, Jean M. and Stanley Presser (1986).
  Survey questions Handcrafting the standardized
  questionnaire. Thousand Oaks, CA Sage
  Publications. Quantitative Applications in the
  Social Sciences Series No. 63.
• Garson, G. David, http//faculty.chass.ncsu.edu/ga
  rson/PA765/survey.htm