Policy Research 101 How to Design a Survey and when and how to avoid it

1
Policy Research 101How to Design a Survey and
when and how to avoid it!

• Keith Curry Lance
• Consultant
• RSL Research Group

2
Major Alternatives to Surveys

• Available data
• Focus group interviews
• Key informant interviews
• Observational studies
• Experiments

3
Key Questions

• What do we need to know?
• How will we use the information?
• Who has it?
• Are they an authoritative source? How willingly
  will they share it?
• What kind of information is it?
• Who, what, how often vs. how, why

4
Issues in Survey Design

• What questions to ask
• General format
• Item formats
• Structured responses
• Wording items
• Ordering items
• Maximizing usable returns

• Format for data entry
• Data entry analysis
• Web-based surveys
• Sampling
• Design
• Administration
• Data processing

5
What Questions to Ask

• Competence of respondent
• Privacy of respondent
• Relevance to issue
• Needing v. wanting to know
• Survey as educational tool

6
General Format

• Length
• White space on page
• Columns, back of page
• Defining sections, spaces emphasizing key items
  (lines, boxes, shading, placement)

7
Item Formats

• Open-ended questions
• Structured responses
• Yes/no, true/false
• multiple choice
• check-off lists
• ranking lists
• Likert scales
• Matrix items

8
Structured Responses

• Standardized ranges
• Comprehensive, exhaustive
• Mutually exclusive

9
Wording Items

• Neutrality, balanced viewpoints
• Clear, brief items
• Simple items (no compounds)
• Redundant items (testing reliability)

10
Ordering of Items

• Logical sequence
• Time sequence
• Controversial item placement
• Halo effect, response sets
• Alternating item format

11
Maximizing Usable Returns

• Keep it short
• Bribery to respond (reward, prizes)
• Easing return (SASE, fold/refold)
• Organizing to receive returns (addressee i.d.,
  storage)
• Follow-up mailings, calls

12
Format for Data Entry

• Numbering cases
• Numbering items
• Placing item numbers
• Spacing responses
• Using column, matrix items
• Alternating item formats
• Capturing numeric responses (check, circle line,
  space, box)

13
Data Entry and Analysis

• Coding alpha, string responses (converting words
  to numbers)
• Missing responses, pages
• Unclear, illegible, inaccurate or multiple
  responses
• Numbering cases
• Spreadsheet, database, data entry, statistical
  software

14
Web-Based Surveys

• Sampling issues
• Design issues
• Administration issues
• Data processing issues

15
Web-Based SurveysSampling

• Difficulty of random sampling
• Self-selection factors
• Sampling biases

16
Web-Based SurveysDesign

• Response format options (radials, toggles,
  drop-downs)
• Controlling responses (single v. multiple
  response items, parameters, contingencies)
• Opportunity for elaboration (contingencies,
  survey length)

17
Web-Based Surveys Administration

• Authentication of respondents
• Follow-up issues
• Web form for data entry from paper

18
Web-Based Surveys Data Processing

• Data delivery options (e-mail, text file,
  database file)
• Data editing options (automatic edit/error checks)

19
Tools for Survey Research

• Links about survey research http//www.lrs.org/re
  sources.aspsurveys
• Links about other research methods
  http//www.lrs.org/resources.asp
• Sample size, cooperation rate, and response rate
  calculators, random date generator, and other
  calculators http//www.lrs.org/tools.asp

20
Contact Information

• Keith Curry Lance
• Consultant, RSL Research Group
• Tel. [phone] - Mobile [phone]
• keithlance_at_comcast.net or
• klance_at_RSLresearch.com
• http//www.linkedin.com/in/keithcurrylance
• http//www.RSLresearch.com