Survey Methods 2

1
Survey Methods 2
Class 20
2
Estimating Sampling Error Assume everything done
correctly thus far Specified sampling
frame Stratified by area Sampled to accommodate
underrepresented groups Is our survey
error-proof? 100 reliable? No Surveyed only
10 of populationsample may misrepresent source
population. How accurate is our sample? Accuracy
is a function of 1) Sample size and 2)
variability of thing measured Sampling error
amount sample differs from total population
3
  Standard Error of the Mean   ?1 SE 67
of sample means   ?2 SE 95 of sample
means   ?2 SE "Confidence interval"     Stand
ard Error SE ?Variance/n  
4
(No Transcript)
5
Standard Errors for Probability   SE for
Probabilities p Probability that the
characteristic exists 1 - p
probability that the characteristic does not
exist p(1-p) variance of
proportion ?p(1-p)/n SE of a
proportion   Example   30 of FIASCO
respondents say that arguments are related to
money. Sample size (n) 1500   SE
?p(1-p)/n ?.30 X .70 /1500 ?.21/1500
.01   Interpretation Confidence interval 2
SE 2 X .01 .02 True mean (proportion) .30
? .02 or 95 confident that true mean lies
between .28 and .32
6
OOptimal Sample Size No Easy AnswersBut a
Possible Rule of Thumb    1. Samples of 150
fairly accurate little gain in precision
beyond n 150   2. Consider the smallest
sub-group you will samplemake sure that that
group is sampled at n 150, as are all other
groups. Then, make sure to weight groups
according to their representativeness.
7
Response Rate
Computation Number Actually Sampled Number in
Targeted Sample   1400 78 Response
Rate 1800   Typical and Acceptable
Ranges  
Office of Management and Budget (OMB) Standard
75   Less than 30 Self selected   Common
response rate Between 25 and 90
8
  Reasons for Non-Response Related to
Bias     Survey Procedure   Respondent Interest
Level   Respondent Education
Level   Respondent Availability   Respondent
Accessibility   Respondent Ability to
Participate Library Digest predicts Alf
Landon for President, 1936
9
Correcting for Non-Response     Proxy
Respondents   OK for factual items (how often
do arguments occur?) Not OK for subjective
items (how do you feel about arguments?)     Stati
stical Adjustments Re-weight sub-group with
high level of non-response (But, what if
non-respondents systematically differ from
respondents?)    Survey Non-respondents (But,
what if responding non-respondents
systematically differ from non-responding
non-respondents?)
10
Developing the Survey Instrument
11
Families in America, Survey of Conflicts
(FIASCO)   1. Sampling frame Nuclear
families S. Orange, NJ   2. Sampling
strategy Systematic stratified (strat. by
quadrants) Differential probability sampling
(oversample new res.)   3. Contact strategy
Random digit dialing (RDD) Only 1 person per
home, parents/kids chance   4. Sample size
n 150 for smallest subgroup, based on est. of
std. error   5. Non-response (NR) strategy
Means to monitor NR Preventive measures
(e.g., ease of response, follow-up calls)
Corrective measures (e.g., stat. adjustments,
survey NRs)
12
Survey Design     1. Define objectives   2.
Draft and refine questions   3. Format
questions   4. Field test instrument
13
Defining Objectives    1. Statement of
Purpose   2. Listing of main variables   3.
Analysis Plan
14
Statement of Purpose FIASCO Survey   The purpose
of this survey is to assess family conflicts in
S. Orange, NJ. Toward this end, the survey
measures the type, frequency, and causes of
conflicts in S. Orange families. The survey also
attempts to relate family characteristics, and
family member's attitudes, towards the type,
frequency, and resolution of conflicts.
  Family characteristics -------gt Conflicts  
Family attitudes -------gt Conflicts
15
Listing of Variables FIASCO Study
Independent Variables
a. Family size
b. Family composition c.
Income level d. Education level e.
______________
Demographics
a. When should conflicts be aired?
b. How should
conflicts be settled?
Attitudes
Dependent Variables
a. Type of conflict
b. Frequency of conflicts
c. Causes of conflicts d. ______________
Conflicts
16
Analysis Plan   1. Sampling Frame (e.g.,
stratified by income)   2. Sampling Method
(e.g., RDD phone survey)   3. Type of
inquiry   a. Associations (correlations)? b.
Causes (experiment)? Pre-post? Between
groups?
17
Criteria for Good Survey Questions
1. Reliable
Respondents understand question meaning
Respondents understand what is expected of them
2. Valid
Question item addresses underlying concept.
18
Road Bumps in Human Communication   (Watzlawick,
Beavin, Jackson, 1967)     Watzlawick
(Vaht-slahv-ick) applied for an assistantship
with a psychiatric research institute.  The
following conversation took place with the
receptionist.
WATZ  Good afternoon, I have an appointment with
Dr. H. My name is Watzlawick   RECEPTIONIST  I
didn't say it was.   WATZ.  (taken aback and
somewhat annoyed).  But I'm telling you it
is.   RECEPTIONIST (bewildered) Why then did you
say it wasn't?   WATZ.  But I said it was.
19
Specific Questions Must Map on to Subjects'
Reality     "Do people in your family employ a
Socratic method of discourse when attempting to
isolate the instigating cause of the discord
under which you collectively toil?"   "To what
degree do people in your household argue about
the influence of the International Monetary Fund
(IMF)?"
20
???-Sorting
21
Q-Sorting
22
??? Groups   Best done with 6-8
people    Types of things participants can
contribute   I. Understanding of
questions a. Concepts b.
Terms II. Insights related to
topic   a. Perceptions b.
Experiences c. Feelings
23
Focus Groups   Best done with 6-8
people    Types of things participants can
contribute   I. Understanding of
questions a. Concepts b.
Terms II. Insights related to
topic   a. Perceptions b.
Experiences c. Feelings
24
FIASCO Focus Group
CONCEPTS AND WORDING   "When do conflicts most
often occur in your family?"   "What do people
in your family most often fight about?"   ___
Finances ____ Chores ____ Noise ___
Messes ____ Other
INSIGHTS   What constitutes a conflict?   What
are the biggest problems with conflicts?   How
do conflicts get started?   How do conflicts
get settled?   How do you feel about
conflicts?   WHAT'S MISSING?
25
Wording Problems   1. Incomplete
wording   2. Inappropriate optional
wording   3. Poor word choice   4. Poorly
defined terms   5. Special cases a.
"Don't Know" b. Wording for special groups
26
Incomplete Wording
Poor form Reason for last conflict?
A 1. Yes
A 2. Because of competition for the TV remote.
Better form   "What was the main reason for the
most recent conflict you personally had with
someone in your family?"
A 1. Because of competition for the TV remote.
27
Unacceptable Optional Wording
Proper optional wording
"You said your last conflict was with your NAME
FAMILY MEMBER How often did you get into
conflicts with (HIM/HER) during the last month?"
Improper optional wording
"How intense was your last conflict" (Was there
shouting, physical threats, acts of violence)?
28
Unclear Meanings
"On what calendared orientation point did the
most recent conflagration transpire amongst those
with whom you share primary genetic bonds?"
PROBLEM?
Overly Complex
"Whom did you last see for professional advice
regarding conflicts in your family?"   "Are you
punished if you don't complete routine chores
(like housecleaning) on time"?
PROBLEM?
Ambiguous Terms
29
Response Format for Open-Ended Questions
" When do arguments most often occur in your
household?"
A 1 When moms tired
A 2 On Sundays
Specify times, quantities, frequencies
Problem?
"What things do you do to calm tensions between
your siblings?"
A 1 A zillion things. 1., 2..., 3.., 4...,
5., 100000000..
A 2 Take a deep breath
Specify the number of points to be made.
Problem?
Why do you and your father argue about money?
A1 Because of rising tuition costs.
A2 Because hes a jerk.
Problem?
Focus the objective of questions.
30
Open Ended and Closed Ended Questions     Open
Ended Where respondent is free to provide an
answer in his or her own words.   "What topics
lead to arguments in your family?"   The things
that lead to fracases in my family are usually
about clothing. Mummy insists that we wear
shirt and ties to the Sunday dinner table, but
Natasha and Sterling just want to wear their
day-glo capes and studded collars.     Closed
Ended Where the respondent is restricted to a
pre-set range of response options.   "What
topics lead to arguments in your family?"   ___
Finances ___ Noise _X_ Decorum (manners,
clothing, make-up) ___ Household chores ___
Opinions about social issues ___ Other
31
Closed Ended vs. Open Ended Questions
Closed Ended Defines question more
clearly Defines response options more
clearly Does not require an interviewer Easy for
respondent Easy for coding/analysis Easy for data
entry
Open Ended
Provides for unexpected responses Less likely to
bias responses Enhances respondent
good-will Historical record
32
(No Transcript)
33
Subjective Response Scales
How would you rate the competence of your
court-appointed family mediator?
To what degree are family conflicts resolved in a
way you find satisfactory?
34
Considerations Regarding Questionnaire Item
Format   1. Limited interpretability of
subjective response scales   2. Agree-disagree
options   3. "Don't Know" option   4. Middle
position choice options   5. Unbalanced vs.
balanced format   6. Measures of attitude
strength
35
How Do You Rate Robeson Cafeteria Food?
36
How Do You Rate Robeson Cafeteria Food?
37
Agree/Disagree Format
Agree
Disagree

1. I wish my family would be more courteous.
2. I am tired of having to watch what I say.
3. Venting pent up feelings is selfish.
4. My family needs to see a conflict mediator.

Problems with this format?
38
Problems With Agree/Disagree Format

• Loose info when squeezing range of attitudes into
  a dichotomy. Cant correlate responses w each
  other.
• Works best for extreme attitudes. Cant
  distinguish moderate or ambivalent atts from
  extreme atts.
• Confusing to indicate a negative (or positive)
  attitude by responding in the affirmative (or
  negative).

Fowler discourages use of agree/disagree for
these reasons.
39
(No Transcript)
40
What Difference Does It Make To Include /
Exclude DK (Dont Know) Response?
1. Does it change rate at which people say
DK?   2. Does it change the rate at which
substantive (non-DK) responses are
endorsed, relative to one another?   3. Does DK
option change relationships between variables?
41
(No Transcript)
42
Solution Provide DK option.
43
Solution Provide DK option.
44
Floaters A Problem and a Mystery     Floaters
People who ??? when there is not a DK (dont
know) filter, but who reply ??? when there
is a DK filter.     Relations between responses
changes when DK filters are used.   Floaters
??? true correlations Floaters can create ???
correls, where none actually exist Floaters
sometimes ???     Who are floaters? ???
45
Floaters A Problem and a Mystery     Floaters
People who give a response when there is not a
DK (dont know) filter, but who reply
Dont Know when there is a DK
filter.     Relations between responses changes
when DK filters are used.   Floaters weaken
true correlations Floaters can create false
correls, where none actually exist Floaters
sometimes make no difference at all     Who are
floaters? Difficult to say. For obscure
questions, floaters tend to be among the less
educated.