Sampling

1
Sampling
2
Goals of Sampling

• Maximize External Validity
• The extent to which the results of a study
  generalize to the population of interest
• To be confident about such a generalization, the
  sample must be representative of the population
  of interest.
• External Validity is NOT Everything.

3
In Defense of External Invalidity (Mook, 1983)

• Sometimes researchers DO NOT necessarily care if
  their findings generalize to real-life behavior
  in the real world
• Sometimes the point is to test theoretically-deriv
  ed hypotheses. Proving it wrong in a non-random
  sample is enough.
• The task is to decide what kinds of conclusions
  that you want to draw from a particular research
  project.

4
Goals of Many Experiments

• Minimize Threats to Internal Validity
• The random assignment of individuals to treatment
  conditions means that confounding variables are
  equally distributed across conditions
• As such, confounding variables are unlikely to be
  responsible for observed differences between
  treatment conditions.

5
Terminology

• Population
• Set of all people, objects, or events of interest
  to the researcher
• Stratum
• A variable that divides the population into
  mutually exclusive segments (e.g., gender, SES,
  political group)
• Population Element
• A single member of the population
• Sample
• A subset of the population used in research

6
Definitions Redux

• A population refers to the aggregate of all of
  the cases that conform to some designated set of
  specifications (Chein, 1981, p. 419)
• The aggregate is the target of generalization.
• A sampling frame is the list that identifies the
  elements of the population.

7
Two Ways to Get a Sample

• Probability Sampling (a) Every element of the
  population has a known nonzero probability of
  being selected (b) Random selection is used at
  some point in the process
• Nonprobability Sampling Something else.
• Bottom Line With nonprobability sampling it is
  NOT possible to estimate sampling errors.
  Moreover, judgments about external validity are
  rarely on firm ground.

8
Probability Sampling

• Any method of sampling that ensures that the
  elements in a population have a KNOWN and CERTAIN
  probability of being chosen. It is not the case
  that all elements MUST have the SAME probability.
  
• Simple Random Sampling All elements have the
  SAME probability of being selected.
• Sampling Frame List or Specification of the
  Population

9
Simple Random Sampling

• All elements have an equal probability of being
  selected.
• Population size is N. Sample size is n.
• If N 1000 and n 100 then the chance that any
  one element would be selected is .10.

10
A Word About Sample Size

• The precision of our estimates increases with
  sample size.
• It is sample size and NOT the size of the
  population that plays the vital role in
  determining precision.
• The function between sample size and precision is
  non-linear.

11
The Standard Error Decreases as Sample Size
Increases (Example SD 10)
12
Another Example of Probability Sampling

• Stratified Random Sampling
• Divide the population into strata and then take a
  simple random sample from each subgroup.
• Sometimes called proportional random sampling
• Here we can over-sample a group if more
  statistical precision is desired for that group.
  This is called disproportionate stratified random
  sampling.

13
Disproportionate Stratified Random Sampling

• We can over-sample particular groups. This way
  we can obtain more precise estimates for strata
  that are small relative to the total population.
• Say there are 100 green people in State X which
  has 10,000 people (Green people are 1 of the
  population). If we really want to know what
  green people are thinking then we should
  over-sample them!
• We must still draw from the entire population of
  green people at random!

14
Nonprobability Sampling

• This method does not involve random selection.
• Lets be Blunt None of these methods are
  terribly good for supporting inferences about
  external validity.
• However, sometimes you cannot use probability
  sampling or do not need to use probability
  sampling.

15
Types of Nonprobability Samples

• Accidental, Haphazard or Convenience Sampling
• The "person on the street" interviews
• College student samples
• Clinical practice samples

16
Purposive Sampling

• One or more specific predefined groups being
  sought. (They are the purpose!)
• Example People in a mall with a clipboard
• Modal Instance Sampling Sampling the most
  frequent case, the "typical" case, or the
  average person
• OK What is the typical or modal person (e.g.,
  the average voter)?

17
Quota Sampling

• Specify the minimum number of sampled units that
  you want in each category.
• Not always concerned with having numbers that
  match the proportions in the population.
  Sometimes, you simply want be able to talk about
  even small groups in the population.

18
Snowball Sampling

• Begin by identifying someone who meets the
  criteria for inclusion in your study
• Then ask them to recommend other potential
  participants who meet inclusion criteria.
• Useful when you are trying to reach populations
  that are inaccessible or hard to find (e.g.,
  homeless)

19
Experience Sampling
20
Sampling of Events

• In addition to sampling people, it is possible to
  sample Events or Experiences
• Examples Age differences in emotional
  experiences Job satisfaction in the workplace
  Motives for drinking
• Often we use technology to help record thoughts,
  feelings, or behavior in the moment.

21
Sampling Strategies

• Sampling at Random
• Signaling device beeps at random periods
  throughout the day.
• Participants are asked to record thoughts,
  feelings, or behaviors.
• Event-Contingent Sampling
• Answer a short questionnaire after an event of
  interest occurs.
• Timing of response is determined by the
  participant

22
Triggers other than Events

• Signaling
• Use an electronic device to signal/remind the
  participants to complete a questionnaire
• Timing Controlled by Researchers
• Daily Diary
• End of day reporting about experiences and
  reactions
• Subject to more biases due to memory
• Does not require electronic devices

23
General Issues

• Brevity Keep it Short
• Frame of reference (at this moment)
• Duration of the Study?
• Common for 7 Days
• 14 Days is probably better
• Possible for much longer intervals

24
Programs for conducting ESP

• ESP The Experience Sampling Program
• A free software package for conducting
  experiments by experience sampling
• Runs on PDAs (Palm Pilots)
• Asks questions of the participant and records the
  answers and the participant's response time.
• The data may later be uploaded to a computer for
  analysis.

25
Experience sampling

• How do you feel right now?
• Please rate each feeling on the scale given. A
  rating of 0 means that you are not experiencing
  that feeling at all. A rating of 6 means that
  this feeling is a very important part of the
  experience. Not at all
  Very Much Happy . . . . . . . . . . . . .
  . . 0 1 2 3 4
  5 6Frustrated/annoyed . . . . 0
  1 2 3 4 5 6

26
A More Efficient Alternative to ESM?

• The research team
• Daniel Kahneman, Princeton University
• Alan Krueger, Princeton University
• David Schkade, University of Texas
• Norbert Schwarz, University of Michigan
• Arthur Stone, Stony Brook University

27
Day Reconstruction Method
28
Sample

• 909 women who worked the previous day
• 49 were European American
• 24 were African American
• 22 were Latinas/Hispanic
• 5 were Other
• Average Age 38 years
• Average Income 54,700

29
Step 1 Construct a Short Diary of the Previous
Day

• Think of your day as a continuous series of
  scenes or episodes in a film. Give each episode
  a brief name. Write down the approximate times
  at which each episode began and ended.
• Average Number 14.1 (SD 4.8)
• Average Duration 61 minutes (Range 15 minutes
  to 120 minutes)

30
Step 2 Answer Structured Questions about Each
Episode
31

• What were you doing? (check all that apply)
• __ commuting __ working
• __ shopping __ preparing food
• __ doing housework __ taking care of your
  children
• __ eating __ pray/worship/meditate
• __ socializing __ watching TV
• __ nap/resting __ computer/internet/email
• __ relaxing __ on the phone
• __ intimate relations __ exercising
• __ other (please specify________________)
•  

32
(No Transcript)
33

•  How did you feel during this episode?Please
  rate each feeling on the scale given. A rating
  of 0 means that you did not experience that
  feeling at all. A rating of 6 means that this
  feeling was a very important part of the
  experience. Please circle the number between 0
  and 6 that best describes how you felt.
•   Not at all Very much
• Impatient for it to end . . . . . . . . . 0
  1 2 3 4 5 6
• Happy . . . . . . . . . . . . . . . . . . . . .
  . 0 1 2 3 4 5
  6
• Frustrated/annoyed . . . . . . . . . . . 0
  1 2 3 4 5 6
• Depressed/blue . . . . . . . . . . . . . . 0
  1 2 3 4 5 6
• Competent/capable . . . . . . . . . . . 0 1
  2 3 4 5 6
• Hassled/pushed around . . . . . . . 0 1
  2 3 4 5 6
• Warm/friendly . . . . . . . . . . . . . . . 0
  1 2 3 4 5 6
• Angry/hostile . . . . . . . . . . . . . . . . 0
  1 2 3 4 5 6
• Worried/anxious . . . . . . . . . . . . . 0
  1 2 3 4 5 6
• Enjoying myself . . . . . . . . . . . . . . 0
  1 2 3 4 5 6
• Criticized/put down . . . . . . . . . . . 0
  1 2 3 4 5 6
• Tired . . . . . . . . . . . . . . . . . . . . . .
  . 0 1 2 3 4 5
  6

34
Comparison of DRM and ESM Affect over Time of Day
ESM
35
Affect Calculation

• Positive Average of Enjoyment, Warm, Happy
• Negative Average of Frustrated, Worried,
  Depressed, Angry, Hassled, Criticized

36
Some Results - Activities
37
Some Results Interaction Partners