Part - time MSc course Epidemiology

1
The following lecture has been approved for
University Undergraduate Students This lecture
may contain information, ideas, concepts and
discursive anecdotes that may be thought
provoking and challenging It is not intended
for the content or delivery to cause
offence Any issues raised in the lecture may
require the viewer to engage in further thought,
insight, reflection or critical evaluation
2
Survey Research Samples and Populations Response
rates Data Types Honesty Delivery
Validity Dr. Craig Jackson Senior Lecturer in
Health Psychology Faculty of Health UCE
Birmingham
3
Brief Research History Fatigue in short-haul
budget airline pilots 2002-2004(Web-based
Survey) Mental Health of UK Farmers using OP
Pesticides (X2) -- 1997-2000(Epidemiological
Surveys) Neurobehavioural Performance of
desert-based Oil Drillers -- 1998-2000(Clinical
assessment) Temporary Hearing Loss in Student
Bar Staff 2000-2002 (Epidemiological
Survey) Benefits of Occupational Health Advice
in Primary Care Settings -- 2001-2004(Randomised
Controlled Trial) Smaller-Scale projects
(Tri-Services, NHS Personnel, NHS
Patients)(Cross-sectional Surveys, Clinical
Trials) Multiple roles of psychologist,
statistician, and methodology designer
4
PatientsStaffHealthy
Quantitative Research Designs
Laboratory
Experimental
RCT
Approach
Case - control
Epidemiology
Cohort study
Observational
Survey
Postal questionnaire
5
Are Postal Surveys Dead Yet? IT predicted death
of postal surveys Use of IT at home and work
increases survey methods Comparison of surveys
using WWW or EMAIL or POSTAL Subjects
UK university staff 200 email
questionnaires 200 emails with www url 100 postal
questionnaires Asked 3 questions teeth
cleaning fruit walking
Jones, R. and Pitt, N. BMJ1999
6
Are Postal Surveys Dead Yet? Results Days after
sending email www post n200 n200 n100
numbers responding 3 25 31 0 5 59 34 1
6 9 61 35 53 10 63 35 60 17 68 37 72 1
7 day response rate 34 19 72 cost
per reply Actual cost 35p 41p 92p With 100
response 19p 7.5p 72p
7
The Postal Survey is Alive and Well . . . .
Communication age? Postal surveys still much
better than e-surveys Novelty value of email is
dead Junk mail perceptions Email filters are
improving fewer emails get through Postal
letters demonstrate emphasize their
importance www surveys email allow immediate
data processing (software) Email www have
potential for low cost regular user
surveys Intranet use of e-surveys seems ok
8
Survey Research Questionnaire is a fundamental
component of most surveys Most MSc / MPhil /PhD
projects use survey methods Can be very
efficient Weaknesses weak / dubious
questionnaires non-valid questionnaires biase
d samples biased responses poor response
rate
9
Wide range of applications Psychometrics --
personality, attitudes, stress, symptoms Physical
measurements -- working, environment,
symptoms Exposure assessments -- hazards, risks,
ppm3, duration, chemicals Necessary
Requirements Take multiple measurements (and
take the mean) Under same conditions, but if
not. Statistical remedies to adjust e.g. age,
time of day etc. Reliable Validated
tool Defined and Regular variables e.g.
problem with stress research Well defined
standard of reference e.g. how far back are
you surveying?
10
Validity Does the survey measure what it says it
is measuring Reliability Does the survey
yield stable data over time
11
Quantitative Research Requires . . . . Numerical
/ Quantifiable data Probability-based Nomothetic
(group data not individual data) Sufficiently
large sample size (to detect statistically
significant effects) Randomised sampling of a
population (cannot guarantee a random
sample) Statistical analyses of data
12
Population Samples
Achieving a high response rate to a questionnaire
is vital as helps ensures a normal distribution
of responses?
Postal questionnaires rarely get a response rate
gt 40 Unless respondents have a vested interest
in the outcome Bias?
Most efficient (best) response rates usually
happen when respondents have to do very little to
take part in the study Multiple phase projects
see a depletion in numbers at every stage Quick
in and out one-stop approach is best
13
RANDOM sampling OPPORTUNISTIC
sampling CONSCRIPTIVE sampling QUOTA
sampling
A Normal Sample
N of population
56 57 58 59
510 511 6 61 62
63 64 Height

14
Being Practical how many is a
sample? Student Pop N indepth I.D Forces
yachting training schools 300 E.M Companies
using stress counselling 150 S.M Divers and ear
barotrauma 142 N.O Solvent exposure in
Myanmar 80 V.W Routine flu vaccinations 900 A
.F Dermatitis in hairdressers 102 S.M O.H needs
of NHS staff 23 yes T.R NIHL in student
employees 14 yes I.C Blood tests in British
Army pilots 408 O.Y Upstream oil company
deaths 161 A.A Renal colic in flight deck
crew 254 A.C Hepatitis B in army regulars and
territorials 476
15

• The Importance of Sample Size
• Forgotten in many studies
• Appropriate size needed to confirm / refute
  hypotheses
• Small samples cant detect anything but the
  grossest difference
• Too large a sample unnecessary waste of
  resources
• Ethical considerations waste of patient time,
  inconvenience, discomfort
• Essential to make assessment of optimal sample
  size before starting investigation

16
(No Transcript)
17
Non-Responders just as Important Postal surveys
may accrue poor response rates (e.g. 20) from
pop. May need to re-write to pop. to re-recruit
bigger sample Inefficient to write to all pop.
again Need to re-write to non-responders and NOT
responders Impossible in anonymous studies with
no linkage Can be done with confidential studies
18
Diminishing returns of multi-stage recruitment
Researcher
Potential Sample
1000 people
540 consents
540 questionnaires
Under-powered studyn 210 Response rate of 21
210 questionnaires
19
Encryption devices Steganography Secret
communication of a message by hiding its
existence Steganos, meaning covered. Gk Graphein,
meaning to write, Gk If message is discovered it
is easily read because of no encryption Cryptogra
phy Secret communication of a message by hiding
its meaning Kryptos, meaning hidden. Gk Message
established using a known protocol, to be
decrypted by the receiver Steganography
Cryptography can be combined together if
needed Steganogrpahy arouses less suspicion in
questionnaire respondents
20
Encryption devices Steganography This example
of steganogrpahy may not work very well when
projected onto a large screen, but it works very
well on paper, such as questionnaires. Hopefully
many of you will not notice the method of
steganography used in this piece of text. Gosh
how clever I am Cryptography drbjh kbdltpo,
jotujuvuf pg pddvqbujpobm ifbmui The above text
(containing an encrypted name and address) looks
suspicious and may be obliterated by the
respondent
drbjh kbdltpo, jotujuvuf pg pddvqbujpobm ifbmui
21
Selection Bias Sampling properly is
Crucial Samples may be askew Specialist
publications attract a specialist response
group Exists a self-selection bias of those with
special interests Controversial topics, or
litigious areas
Bird Flu
Call Centres
Gulf War Syndrome
THIS IS AN INHERENT PROBLEM WITH HEALTH RESEARCH
COMBAT IT WITH LARGE SAMPLES AND CLEVER
METHODOLOGY
Depleted Uranium Weaponry
Organophosphate Pesticides
Stress
Telecomms
22
Structure of Surveys Identifying Items
Title Preamble Instructions Research
Items does not need to be honest some
deception is necessary shorter surveys better
than long surveys
23
Identifying items Demographics Preliminary
questions Collecting info necessary for
screening recording keeping tracking tracing
data manipulation Ask only for relevant info
unethical to harvest data Fewer items
minimizes chance of alienating respondent
Subjects need for privacy anonymity
The need for rich info to improve the study
24
Preamble An important introduction Frame of
reference Without a FOR a respondent may base
their answer on a wrong context We would like
to know if you have had any medical complaints
and how your health has been in general, over the
past few weeks. Please answer ALL the questions
on the following pages by ticking the answer
which best applies to you. Remember that we want
to know about present and recent complaints, not
those that you had in the past. It is important
that you try to answer all questions
25
Title Not to be underestimated try to be
user-friendly Sets the tone for the
respondent The General Health Questionnaire vs
The GHQ 28 Beware of abbreviated titles - may
alienate respondents e.g. ASS - Agricultural
Satisfaction Scale LEI Life Events
Inventory BDI Beck Depression Inventory MMPI
Minnesota Multiphasic Personality
Inventory HAD Hospital Anxiety Depression
scale
Simple language for simple people
26
(No Transcript)
27
Are Respondents Telling the Truth? Use a
sprinkling of LIE DETECTOR items to assess
reliability e.g Have you ever taken anything
without asking permission? Did you ever lie to
your parents as a child? Have you ever visited
a pornographic web-site? Decide what to do
with any respondent who fails the lie
detector Keep them in? Exclude
immediately? Retain but flag them?
28
Response-Set Bias Enables collection of detailed
data easily Build a detailed data set from a
simple binary item Randomise the layout / order
of responses true / false e.g I eat ice
cream True False I like ice cream
True False I enjoy ice cream True False I love
ice cream True False I hate ice cream
True False
Respondents ticking the same items Swap the
True False responses sequence
29
Likert scales A visual linear scale for rating
purposes
Swap around the words to avoid response-set
bias Avoid using numbers on the scale can be
(mis)leading
30
Likert scales
31
Likert scales Bend your data and avoid
fence-sitters
X
X
X
X
X
X
X
X
X
X
X
X
32
Design Pointers for Surveys Avoid colloquialisms
or abbreviations Beware of local
expressions Avoid words with double meanings
e.g.Fair....Dip....Lie....Well Set a
definition of specific terms e.g.
OK.....Average Avoid long questions Specify
exact time, place and context e.g At
school, did you ever... Avoid impossible denial
e.g When did you first... Avoid numbers on
your responses / scales Have items seemingly
related to the research topic
33
Design Pointers for Surveys Clearly phrase
items Make items unambiguous Avoid leading
items e.g. Many people think... Ask only
what a respondent is qualified to answer Avoid
socially loaded items e.g. beggars or
junkies Do not use socially / religiously
biased items Avoid phrases, clichés or
sooth-sayings Ask for ages as a true number /
D.O.B - - NOT in age groups Easy
Fonts Pictures (not smiley faces!)
34
Some Common Errors in Surveys
In your organization, do women have the same
responsibilities as men, and should women have
more?
Preventing accidents in the workplace is vital,
and more money should be spent on prevention.
Training in risk assessment is something I would
like to do, and I would like to see my colleagues
do it to.
Only ask one question per item
35
Some Common Errors in Surveys
What is your age ? (please tick) 16 - 26 26 -
36 36 - 46 46 - 56
What is your marital status ? (please
tick) Married Single
How many blood splashes have you had ? (please
tick) 1 - 5 6 - 10 11 - 15 15
- 20 20 plus
36
Factorial approach The General Health
Questionnaire (GHQ 28) A self-completion
questionnaire assessing mental health How? 28
items 7 about Anxiety Anxiety score 3 7 about
Severe depression Depression score 4 7 about
Dysfunction Dysfunction score 5 7 about Somatic
symptoms Somatic score 2 Global score
14 By summing the 4 factors there is a
Global Mental Health score Statistics are
performed on the factor scores and the global
score NOT on each individual item
37
Increasing Response Rates Incentives Appearance
Delivery Origin Contact Content Communication

Edwards et al. 2002
38
Incentives Money Vouchers Prize draw Ethical
aspects Bias O.R Monetary incentive
vs. None 2.02 Incentive with Q. vs. Incentive
on return 1.71 Non-monetary incentive vs. No
incentive 1.19
Edwards et al. 2002
39
Appearance O.R Shorter format vs. Longer
format 1.86 Brown envelope vs.
White 1.52 Coloured ink vs. Black 1.39 Fold
er / Booklet vs. Stapled pages 1.17 Personalised
vs. Not personalised 1.16 ID feature on return
vs. No ID 1.08 Coloured Q vs. White Q 1.06
Edwards et al. 2002
40
Delivery methods O.R Recorded delivery
vs. Standard 2.21 Stamped return envelope vs.
Business reply / franked 1.26 Q sent to work vs.
Q sent to home 1.16 1st class outward mail vs.
Other class 1.12 Pre-paid return envelope vs.
Not pre-paid 1.09 Stamped outward envelope vs.
Franked 0.95 Commemorative stamp vs. Ordinary
stamp 0.92
Edwards et al. 2002
41
Origin Contact Origin O.R University vs.
Other organisation 1.31 Sent by senior persons
vs. Juniors 1.13 Ethnically ambiguous name vs.
Non-white name 1.11 Contact O.R Pre-contac
t vs. No contact 1.54 Follow up vs. No follow
up 1.44 Postal follow up with Q vs. Without
Q 1.41 Mentioning follow up vs.
None 1.04 Pre-contact by telephone vs. Postal
pre-contact 0.90
Edwards et al. 2002
42
Content O.R More interesting vs. Less
interesting 2.44 User-friendly vs.
Standard 1.46 Factual items only vs. Factual
attitude items 1.34 Relevant items first vs.
Other items 1.23 Demographic items first vs.
Other items 1.04 Dont know boxes vs. no
Dont know boxes 1.03 Sensitive items vs. No
sensitive items 0.92 General items first vs.
Last 0.80
Edwards et al. 2002
43
Communication O.R Explain drop-out
required vs. Not 1.32 Stresses benefit to
respondent vs. Others 1.06 Stresses benefit to
sponsors vs. Others 1.01 Stresses benefit to
society vs. Others 1.00 Response deadline given
vs. No deadline 1.00 Instructions given vs. No
instructions 0.89 Choice to opt out given vs.
No opt out 0.76
Edwards et al. 2002
44
Human Nature Then . . . . . Cash incentives
work Want cash upfront not contingent Like to
feel important Little time to spare Not motivated
by benefit to any other people Lazy
45

• Survey Summary
• Surveys widely used in data collection
• Mark pre-pay or addressed envelopes
• Steganography better than cryptography
• Perform analyses on factors not individual items
• Identifying items / demographics usually useful
  in later analyses
• Think about scoring before rushing ahead with
  survey
• Store questionnaires securely until passing any
  viva voce / publication
• Collect as much info at source, parse it down
  later at discretion

46

• Survey Summary
• Perfect for epidemiological studies and health
  research
• Non-response to postal questionnaires reduces
  sample introduces bias
• Identification of effective ways to increase
  postal response rates
• Use existing metrics - pilot items if making new
  questionnaire
• Use factor analysis
• Keep it all as brief as possible
• Dont alienate respondents
• Alternate types of items

47
Final Points Bias does not ruin
everything Bias Avoiding bias is a good aim to
have Existence of some bias in a sample does not
ruin a project entirely Mostly leads to
underestimation rather than overestimation of
main effects Spector et al, (2000) shows the
inflating effect of self-report bias may not be
so prominent Spector PE, Chen PY, OConnell BJ.
A longitudinal study of relations between job
stressors and job strains while controlling for
prior negative affectivity and strains. Journal
of Applied Psychology 2000 85 211-218.
48
Final Points Generalizability is not
everything Basic principle Internal validity
is always more important than its
generalizability Never appropriate to
generalise an invalid finding Mant et al.
(1996) Mant J, Dawes M, Graham-Jones S.
Internal validity of trials is more important
than generalizability. British Medical Journal
1996 312 779.