SURVEY RESEARCH

1
SURVEY RESEARCH

• TTM531 Fall 2003 Class Presentation
• Nancy Whelchel
• Coordinator for Survey Research
• University Planning and Analysis
• NCSU
• nancy_whelchel_at_ncsu.edu
• 515-4184
• http//www2.acs.ncsu.edu/UPA/survey/presentations/
  ttm531.survey_research.ppt

2
What is a Survey?

• A scientific social research method that involves
• Selecting a random sample of people to answer
  some questions
• Designing a standardized questionnaire to get
  information about the research question(s)
• Administering the questionnaire
• Coding the responses in a standardized form

3
What is a Survey?

• continued
• Analyzing the results to provide descriptions
  about the people in the sample and find
  relationships between different responses
• Generalizing the results to the population from
  which the sample was drawn
• Earl Babbie (1990). Survey Research Methods, 2nd
  ed. California Wadsworth Publishing Company.

4
Why Do a Survey?

• Bad Answer
• My boss told me to.
• Better Answer
• I want to systematically evaluate the extent to
  which my program is meeting its objectives and
  outcomes in order to improve our programs through
  pro-active, well-informed decision making.
• Adapted from CUPR FAQ What is Assessment.
  (http//www.ncsu.edu/provost/
• academic_programs/uapr/FAQ/UAPRFAQwhatisassessment
  .html)

5
But Why Do a Survey?

• To collect information that is not available from
  other sources
• Attitudes and opinions
• E.G., Employee attitudes about telecommunting
• Behavior
• E.G., Number of hours employees working at home
• Using a standardized measurement (questionnaire)
  provides comparable information from everyone
  taking the survey, which allows for meaningful
  analysis

6
But Why Do a Survey?

• Continued
• With a good sample, you can have confidence that
  the results can be generalized to the population
  in which you are interested
• Findings are not simply based on the opinions of
  those who speak up at meetings
• Floyd J. Fowler, Jr. (1988). Survey Research
  Methods. Beverly Hills, CA Sage Publications

7
Before You Plan to do a Survey...

• Think About Available Resources
• What information is already available from
  non-survey sources?
• Have any surveys already been done on this
  question?

8
Before You Plan to do a Survey...

• Think About Who is going to
• Design the survey
• Administer the survey
• Enter the data
• Analyze and interpret the data
• Write up the results
• Present the findings
• Use the findings
• Pay for it all...

9

• So, you still want to do a survey

10
Major Tasks

• ID Purpose of Research
• Decide on Survey Design and Format
• Select Sample
• Design Survey Instrument
• Pre-test Instrument
• Collect Data
• Analyze Data
• Report Findings

11
Purpose of Your Survey

• Ask Yourself
• WHY do I want to/need to do a survey?
• WHAT do I want to learn?
• WHO is my audience?
• HOW will I use the information I gather?

12
Purpose of Your Survey

• Our Example
• Why
• President of NCTextiles.com, where you work, told
  you to.
• What want to learn
• How does employee satisfaction and productivity
  compare between NCTextiles.com telecommuters and
  non-telecommuters.
• Who is audience
• Company president and executive board.
• How use
• Set new policies for telecommuting.

13
Purpose of Your Survey

• Description
• Discover the distribution of certain traits
  or characteristics
• (e.g., telecommuters, revenue generated)
• Explanation
• Understand why something happens
• (e.g., is there a difference between
  telecommuters and non-telecommuters in job
  satisfaction and in productivity)
• Exploration
• Help inform a research design

14
Basic Survey Designs

• Cross-sectional
• Data collected at one point in time from a sample
  selected to describe a larger population
• Longitudinal
• Trend
• Different samples from a general population
  studied at different points in time.
• Cohort
• Same specific population studied over time (may
  be different samples).
• Panel
• Same sample studied over time.

15
Basic Survey Formats

• Format determines much of research design (e.g.,
  sampling, questionnaire design, administration,
  data analysis, etc.)
• Basic formats include
• Self-administered mail survey
• Telephone interview
• Face-to-face interview
• Focus group
• Self-administered Web survey
• Self-administered Email survey

16
The Survey Sample

• WHO or WHAT do I want to survey?
• Sampling allows you to study just a portion of
  the population/group and make estimated
  assertions about the population from the sample.
• Warning Bad Sample Bad Data

17
Why Sample?

• Time
• Shorter time in the field decreases the chance
  that external factors will influence results.
• Quality
• Doing fewer surveys increases control over the
  entire process (interviewing, data entry, etc.)
  and therefore decreases error
• Cost

18
What is a Sample?

• A Warning
• Findings from a survey are only representative of
  the aggregate of elements in the sampling frame.
• In other words
• You cant use your results to say anything about
  the folks who didnt have a chance to be included.

19
Sample Size

• Depends on
• Cost
• Level of precision want (margin of error)
• Level of confidence want
• Anticipated response rate
• Number of responses needed per subgroup
  interested in

20
Sample Size

• Example Margin of Error
• /- points confidence interval
• 80 90 95
• 5 150 275 400
• 10 40 70 100
• If 45 of 400 respondents agree that
  NCTextiles.com provides excellent health
  benefits, this means that 95 out of 100 times
  between 40 (45-5) and 50 (455) believe the
  company provides excellent health benefits.
• Source McClelland, Lou (1993), Survey Research
  Mechanics in Three Checklists, (working paper).
  Office of Research and Information University of
  Colorado at Boulder.

21
Selecting a Sample

• Probability Sample
• A representative sample
• Everyone in the sample population has an equal
  chance of getting in the sample.
• Results can be generalized to the sample
  population.
• Non-Probability Sample
• Non-representative sample
• Purposive, quota, whos available
• Results can not be generalized beyond the sample.

22
Probability Sampling

• Simple random sample
• List the elements in the sampling frame and use a
  random number chart to select elements.
• Systematic sampling
• List the elements in the sampling frame and take
  every Nth element, based on how many needed in
  sample.
• Warning Be attentive to any inherent patterns in
  how sampling frame is ordered.

23
Probability Sampling

• Stratified sampling
• Draw appropriate proportion of elements from
  homogeneous groups interested in (based on
  theory).
• Note Think about using disproportionate
  sampling then weighting the data. E.G., If
  there are relatively few numbers in one sub-group
  you are interested in, include proportionally
  more of them in your sample, then weight the
  data to reflect their true proportion in your
  population.

24
Probability Sampling

• Stratified sampling continued
• An Example
• 250 out of your 2000 employees in the sample
  population telecommute (12.5 of sample
  population).
• With stratified sampling with sample size of 800,
  would have 100 telecommuters (800 .125), and
  700 non-telecommuters (800 .875).

25
Probability Sampling

• Stratified sampling continued
• An Example continued
• BUT, you might want to oversample telecommuters
  (even include all 250) to be able have less error
  in their results, and undersample
  non-telecommuters (e.g., 550). When analyzing
  all results, weight down telecommuters to
  reflect their true proportion in the sample
  population, and weight up non-telecommuters.

26
Instrument Design

• Start with
• Conceptualization
• Define the concepts you want to study
• Operationalization
• Specify indicators of your concepts.
• Make exhaustive list of everything the concept
  could include

27
Instrument Design

• Conceptualization
• Job Satisfaction
• A pleasurable or positive emotional state
  resulting from the appraisal of ones job.
• Includes both extrinsic and intrinsic aspects.
• Has several dimensions
• importance to employee
• motivation of employee
• current level of attainment
• satisfaction with attainment

28
Instrument Design
Operationalization Job Statisfaction

• Extrinsic
• pay
• promotion
• supervision
• co-workers
• company
• physical conditions
• security
• social status
• work itself

• Intrinsic
• variety of work
• autonomy and freedom
• responsibility
• feedback
• creativity
• recognition
• meaningfulness of work

29
Questionnaire Construction

• The way a question or statement is worded and the
  response options offered determine the nature of
  the data received.
• Bad question Bad data

30
Questionnaire Construction

• To have confidence in your survey results your
  questions must be
• RELIABLE
• They provide consistent measures in comparable
  situations.
• AND
• VALID
• They measure what they are intended to measure.

31
Questionnaire Construction

• Open-end Questions
• Respondents write question response in own words
• probably need to enter data by hand
• develop coding scheme for responses
• content analysis ?
• Suggestion Its a good idea to always include
  an open-end question giving the respondent the
  opportunity to add any additional comments they
  might have.

32
Questionnaire Construction

• Closed-end or Forced-choice Questions
• Respondent selects question answer from among a
  list of possible responses
• Suggestion Its a good idea to ask multiple
  indicators of a single concept, and then create a
  scale by combining those indicators.

33
Good Questions...

• Good Questions...
• Are clear and use simple language
• Are concise
• Are specific
• Are possible to answer
• Are relevant to the respondent
• Do not use negatives
• Avoid biased terms
• Have only 1 part (not double-barrel)

34
Good Questions...

• Use simple language...
• My job is usually interesting enough to keep me
  from getting bored.
• Is better than
• My job is usually engaging enough to permit me
  from reaching a banausic state of mind.

35
Good Questions...

• Are concise...
• How interested would you be in a flexible work
  schedule in which you would have to work an 8
  hour day in the office, but would be able to
  arrive at work anytime between 600 and 1000 AM
  and leave between 200 and 700?.
• Is better than
• How interested would you be in being able to
  leave your house either before the morning rush
  hour or after the morning rush hour, and then
  arrive at the office not at the standard 800 AM
  time that is required according to the employee
  handbook, but at whatever time you got there, and
  then work a 8 hour day at the office (excluding
  and hour for lunch), and then leave the office so
  you are not commuting during rush hour but either
  before or after rush hour?

36
Good Questions...

• Are specific...
• In what year did you first start working at
  NCTextiles.com?
• Is better than
• When did you start working here?

37
Good Questions...

• Are possible for the respondent to answer...
• NO
• My contribution to the company has had a direct
  impact on its ability to control a broader share
  in the global market.

38
Good Questions...

• Are relevant...
• NO
• Polyester doesnt breathe as well as natural
  fibers.

39
Good Questions...

• Do not use negatives...
• The lack of flexibility in my work hours makes it
  difficult for me to spend the amount of time I
  would like to with my family.
• Is better than
• The lack of flexibility in my work hours does not
  make it possible for me to spend the amount of
  time I would like to with my family.

40
Good Questions...

• Do not use biased terms or suggestions...
• Do you agree or disagree with the following
  statement
• All employees need to be in the office during
  regular business hours.
• Is better than
• Dont you agree with the president of the company
  when she says that all employees need to be in
  the office during regular business hours?

41
Good Questions..

• Do not ask two questions in one (double-
  barrel)...
• I find real enjoyment in the work I do.
• I like the hours I keep at work.
• Is better than
• I find real enjoyment in the work I do and the
  hours I keep.

42
Response Options

• Response options should reflect the concepts you
  are trying to measure, and fit with the question
  wording.
• Suggestion Avoid simple yes or no type
  responses and try to measure intensity of feeling
  when possible.

43
Response Options

• Must be
• Mutually exclusive
• can select only one appropriate answer
• Example NOT mutually exclusive...
• Q. On most days, what time do you leave your
  house for work?
• 600 AM or earlier 730-800 AM
• 600-630 AM 800-830 AM
• 630-700 AM 830-900 AM
• 700-730 AM 900 AM or
  later

44
Response Options

• Mutually exclusive continued
• Example IS mutually exclusive...
• Q. On most days, what time do you leave your
  house for work?
• Before 600 AM 730-759 AM
• 600-629 AM 800-829 AM
• 630-659 AM 830-859 AM
• 700-729 AM 900 AM or
  later

45
Response Options

• Must be
• Exhaustive
• all possible answers are listed (including e.g.,
  other, dont know, etc.)
• Example NOT exhaustive
• Q. On most days, what time do you leave your
  house for work?
• 600-629 AM 730-759 AM
• 630-659 AM 800-829 AM
• 700-729 AM 830-859 AM

46
Response Options

• Exhaustive continued
• Example IS exhaustive
• Q. On most days, what time do you leave your
  house for work?
• Before 600 AM 730-759 AM
• 600-629 AM 800-829 AM
• 630-659 AM 830-859 AM
• 700-729 AM 900 AM or
  later

47
Questionnaire Format

• Helpful hint.
• Be sure to seek advice on question wording,
  response options, and questionnaire format from
  those who will be doing the data entry and data
  analyses.

48
Questionnaire Format

• Should be visually attractive and nicely
  reproduced
• Should be spread out and uncluttered
• better to have more than 1 page than cramped page
• Should be broken into logical sections when
  possible
• Should have clear skip patterns for contingency
  questions

49
Questionnaire Format

• Should have clear spaces for respondents to mark
  answers
• use boxes
• parentheses ( X )
• or numbers to circle
• avoid lines to put check on X_

X
1
50
Question Order

• Be attentive Initial questions effect answers to
  subsequent ones.
• Ask less interesting questions (e.g.,
  demographics) at end of self-administered survey
  and beginning of interview survey.

51
Instructions for Respondent

• Provide general instructions in beginning of
  self-administered survey
• Brief explanation could include
• why doing survey
• why answers are important
• how to answer questions
• stress confidentiality (if appropriate)
• Do NOT ask permission - - assume respondent will
  want to participate.
• Provide specific instructions to questions as
  necessary

52
Instructions for Respondent

• Example
• Instructions The following statements are about
  some features of your job. Please tell us how
  satisfied or dissatisfied you feel with the
  features of your present job by circling one of
  the response options listed below each statement.
• There are no right or wrong answers. All replies
  are confidential so give your honest opinion on
  each of the statements. Please do not discuss
  your answers with other people. It is your
  opinion which matters.
• Even though it may be hard to decide, please be
  sure to answer all questions. Thank you for your
  help.

53
Pre-Test

• Test the survey instrument before doing the
  actual study.
• Suggestion Its also a good idea to test out
  your sample design, data collection methods, data
  processing, and data analysis if possible.

54
Pre-Testing the Survey Instrument

• Use respondents similar to those who will be in
  survey sample.
• Test
• Question clarity
• failure to answer?
• multiple answers?
• other answers?
• qualified answers?

55
Pre-Testing the Survey Instrument

• Test continued
• Questionnaire format
• instructions?
• flow?
• layout?
• Variance in responses (if interested in
  relationships among variables)
• do most respondents give the same answer to a
  question?

56
Pre-Testing the Survey Instrument

• Revise survey instrument as necessary
• Pre-test again if possible
• (ideally with some of original pre-test
  respondents, and some new respondents)

57
Data Collection(Self-administered mail survey)

• Include
• Survey instrument
• Return envelope
• self-addressed and stamped
• (either postage-paid or 1st class stamp)
• Cover letter
• on letterhead paper
• signed by most important person possible

58
Data Collection(Self-administered mail survey)

• Cover letter (continued)
• brief explanation of
• purpose of survey
• how results will be used
• why respondent was selected
• why all answers are important
• emphasize confidentiality (if applicable)
• provide contact information for respondents with
  questions about the survey
• Describe/explain any incentives for participating
  in the survey and/or consequences for
  non-participation if applicable.

59
Identifying Respondents and Non-Respondents

• If respondents are not asked to provide any
  identifying information on the survey, you can
• Number the surveys before they are sent out
• OR
• Unobtrusively number the return envelopes before
  they are sent in the survey packet
• AND
• Keep a list of survey/envelope numbers
  corresponding to respondent name or id.

60
Monitor Returned Surveys

• Open surveys as they are returned
• Assign unique ID number to each survey (if not
  already assigned)
• Stamp date on survey
• Track the number of survey coming in each day
• Use this information to help inform timing of
  follow-up reminders.

61
Follow-up Reminders

• Sending 2 follow-up reminders works best to
  increase response rates.
• Timing
• first follow-up 2-3 weeks after original
• second follow-up 2-3 weeks after 1st follow-up
• Materials
• reminder card with information to have new survey
  sent if necessary OR
• entire survey packet

62
Follow-up Reminders

• Send to
• Non-respondents only if possible to identify
• All respondents if not possible to identify
  non-respondents
• include if you have already responded, thank you
  for participating.

63
Response Rates

• Response Rate
• n surveys completed and returned
• net sample size
• (net sample size sample size - undeliverable
  surveys)
• In general
• Acceptable response rate 50
• Good response rate 60
• Very good response rate 70

64
And then.

• Data Entry
• Data Analysis
• Report Writing
• Presentation

65
Some References

• Babbie, Earl (2002), The Basics of Social
  Research 2nd ed. Wadsworth Thomson Learning CA.
• Dillman, D.A. (2000), Mail and Internet Surveys
  2nd ed. Wiley NY.
• Fink, Arlene and Jacqueline Kosecoff (1998), How
  to Conduct Surveys. Sage CA.
• Fowler, Floyd J. (1993), Survey Research
  Methods. Sage CA.
• Fowler, Floyd J. (1995), Improving Survey
  Questions. Sage CA.
• Sudman, Seymour and Bradburn, Norman (1982), A
  Practical Guide to Questionnaire Design.
  Jossey-Bass San Francisco.
• Tourangeau, R., and Smith, T.W. (1996), Asking
  Senstive Questions The Impact of Data Collection
  Mode, Question Format, and Question Context,
  Public Opinion Quarterly, 60275-304.
• Also see American Association of Public Opinion
  Research, Best Practices for Survey and Public
  Opinion Research. http//www.aapor.org/ethics/be
  st.html