NonExperimental designs: Surveys

1
Non-Experimental designs Surveys
Quasi-Experiments

• Psych 231 Research Methods in Psychology

2
Announcements

• Lab attendance is critical this week because
  group projects are being administered
• Attendance will be taken.
• Turn in the group project rating sheet 1

3
Non-Experimental designs

• Sometimes you just cant perform a fully
  controlled experiment
• Because of the issue of interest
• Limited resources (not enough subjects,
  observations are too costly, etc).
• Surveys
• Quasi-Experiments
• Developmental designs
• Small-N designs
• This does NOT imply that they are bad designs
• Just remember the advantages and disadvantages of
  each

4
Surveys

• What are they?
• Questionnaires and interviews that ask people to
  provide information about themselves
• Why conduct them?
• To compliment experimental work
• Good/common first step, can collect a lot of data
  about a lot of variables
• Do not have to directly observe behaviors
• Best way to collect some kinds of information
• Descriptive, behavioral, and preferential
• (e.g. demographic information, recreational
  behavior, and attitudes)

5
Surveys

• Advantages
• One can investigate internal events (for example,
  attitudes opinions)
• Can generalize about an entire population based
  on relatively small samples of individuals
• Large amounts of data can be collected quickly
  with relatively little cost (effort, time, etc.)
• But theyre often not as cheap as you may think

6
Surveys

• Disadvantages
• Correlational causal claims shouldnt be made
• Non-response bias
• Why doesnt everybody respond?
• Does response rate interact with variables of
  interest?
• Large data sets are sometimes difficult to
  analyze
• Self-reports may not be truthful
• Response set - tendency to respond from a
  particular perspective (e.g., how a moral
  person would answer)

7
Stages of survey research

• Stage 1) Identify the focus of the study and
  select your research method
• What are the objectives of the research?
• Is a survey method the best approach?
• What kind of survey should be used?

8
Surveys methods

• Many different methods are used to administer
  surveys

• Group administration (e.g. MASS testing session)
• Mail surveys
• Internet surveys
• Telephone surveys
• Face-to-face interviews
• Focus group interviews

9
Stages of survey research cont.

• Stage 2) Determining the research schedule and
  budget
• Stage 3) Establishing an information base
• Find out whats been done, whats known
• E.g., Find other related surveys
• Stage 4) Identify the sampling frame
• The actual population that the sample is drawn
  from (as opposed to the ideal population)
• Think of it as operationalizing the conceptual
  level population

10
Stages of survey research cont.

• Stage 5) Determining the sample method and
  sampling size
• Review Probability and Non-Probability methods
• Voluntary response method
• Importance of sample size

11
Voluntary response methods

• A kind of convenience sampling methods commonly
  used

• Problem You typically get only individuals with
  strong opinions to respond, so the results are
  often extremely biased

12
Importance of sample size

• Sampling error - how is the sample different from
  the population?
• Confidence intervals
• An estimate of where the mean or percentage in
  the overall population is, based on the sample
  data
• John Doe has 55 of the vote, with a margin of
  error 3
• Margin of error (that 3 part)
• Which would you be more likely to believe
• We asked 10 people
• We asked 1000 people
• The larger your sample size, the smaller your
  margin of error will be.

13
Stages of survey research cont.

• Stage 6) Designing the survey instrument
• Question construction How the questions are
  written is very important
• Clearly identify the research objectives
• Do your questions really target those research
  objectives?
• Take care wording of the questions
• Keep it simple, dont ask two things at once,
  avoid loaded or biased questions, etc.
• How should questions be answered?

14
Good and poor questions

• Good

Poor

• Was the FDC negligent
• by ignoring the warnings
• about Vioxx during testing
• and approving it for sale?
• Yes
• No
• Unsure

• If the FDC knew that
• Vioxx caused serious
• side effects during testing,
• what should it have done?
• Ban it from ever being sold
• Require more testing before approving it
• Unsure

15
Good and poor questions

• Good

Poor

• Are you against same sex
• marriage and in favor of a
• constitutional amendment
• to ban it?
• Yes
• No
• Unsure

• What is your view on same sex marriage?
• I think marriage is a matter of personal choice
• Im against it but dont want a constitutional
  amendment
• I want a constitutional amendment banning it

Problem Biased in more than one direction
16
Survey Questions

• Question types
• Open-ended (fill in the blank, short answer)
• Can get a lot of information, but
• Coding is time intensive and potentially
  ambiguous
• Close-ended (pick best answer, pick all that
  apply)
• Easier to code
• Response alternatives are the same for everyone
• Rating scales
• Used for how much judgments
• e.g., Likert scale measures attitudes,
  agree/disagree
• Take care with your labels
• Range of scores, anchors

17
Stages of survey research cont.

• Stage 7) Pre-testing the survey instrument
• Fix what doesnt seem to be working
• Stage 8) Selecting and training interviewers
• For telephone and in-person surveys
• Need to avoid interviewer bias
• Stage 9) Implementing the survey
• Stage 10) Coding and entering the data
• Stage 11) Analyzing the data and preparing a
  final report

18
Error in survey research

• Measurement error
• Sampling error

19
Error in survey research

• Measurement error
• Same old issues of validity and reliability
• Are your questions really measuring what you want
  them to?
• Do you get similar answers with repeated questions

20
Error in survey research

• Measurement error

• Sampling error

• Are there differences in your sample (compared to
  the population as a whole)?
• Response rate
• What proportion of the sample actually responded
  to the survey?
• Hidden costs here - what can you do to increase
  response rates
• Non-response error (bias)
• Is there something special about the data that
  youre missing? From the people who didnt
  respond

21
Quasi-experiments

• What are they?
• Almost true experiments, but with an inherent
  confounding variable
• General types
• An event occurs that the experimenter doesnt
  manipulate
• Something not under the experimenters control
• (e.g., flashbulb memories for traumatic events)
• Interested in subject variables
• high vs. low IQ, males vs. females
• Time is used as a variable

22
Quasi-experiments

• Advantages
• Allows applied research when experiments not
  possible
• Threats to internal validity can be assessed
  (sometimes)

23
Quasi-experiments

• Disadvantages
• Threats to internal validity may exist
• Designs are more complex than traditional
  experiments
• Statistical analysis can be difficult
• Most statistical analyses assume randomness

24
Quasi-experiments

• Program evaluation
• Research on programs that is implemented to
  achieve some positive effect on a group of
  individuals.
• e.g., does abstinence from sex program work in
  schools
• Steps in program evaluation
• Needs assessment - is there a problem?
• Program theory assessment - does program address
  the needs?
• Process evaluation - does it reach the target
  population? Is it being run correctly?
• Outcome evaluation - are the intended outcomes
  being realized?
• Efficiency assessment- was it worth it? The the
  benefits worth the costs?

25
Quasi-experiments

• Nonequivalent control group designs
• with pretest and posttest (most common)
• (think back to the second control lecture)

• But remember that the results may be
  compromised because of the nonequivalent control
  group (review threats to internal validity)

26
Quasi-experiments

• Interrupted time series designs
• Observe a single group multiple times prior to
  and after a treatment
• Obs Obs Obs Obs Treatment Obs Obs Obs Obs
• Look for an instantaneous, permanent change
• Variations of basic time series design
• Addition of a nonequivalent no-treatment control
  group time series
• O O O T O O O O O O _ O O O
• Interrupted time series with removed treatment
• If treatment effect is reversible

27
Next time

• Go to labs this week, attendance will be taken
• Non experimental designs cont.
• Reminder, journal summary 2 is coming up