FORMULATING THE QUESTION PRICNIPLES OF STUDY DESIGN

1
FORMULATING THE QUESTION PRICNIPLES OF STUDY
DESIGN

• Marion Campbell

2
Purpose of Health Services Research

• Health services research is the discipline
  which seeks knowledge will lead to improvements
  in the delivery of health care.
• Crombie IL (1996)

3
Desirable Attributesof Study Design

• Clear definition of the study question
• Precise estimate of variables/outcomes of
  interest
• Reliable and valid estimation of likely effects
  of alternative interventions without systematic
  bias
• Observed effects attributable to intervention
• Observed effects generalisable to other contexts

4

• DEFINING THE QUESTION

5
Typical scenario

• A clinician/NHS manager will come to you with an
  idea for a new intervention/service. They want
  you to help them decide what the impact of this
  new development might be.
• You then need to change this vague concept into a
  researchable question

6
The PICOT method

• P - Patient population who is to receive the
  intervention?
• I Intervention - What treatment do you need to
  test the effect of?
• C Comparator - What treatment are you comparing
  against (could be alternative treatment or no
  treatment)?
• O Outcome - How is the effect of the
  intervention measured?
• T- Target - What is the target of the study? ie.
  are you asking whether the new intervention is
  'superior' to the alternative, or 'non-inferior'?

7
Example

• NHS manager wants to explore the impact of
  hormone replacement therapy
• P population - peri-menopausal women
• I intervention - hormone replacement patches
• C comparator - No treatment
• O outcome - Effect on Bone density
• T target superiority study

8
HSR question

• Original
• What is the impact of HRT?
• Revised
• Does the provision of HRT patches to
  peri-menopausal women result in increased bone
  density when compared with a policy of no
  treatment?

9
The truth is out there!

• Aim of research study is to identify the Truth
  ie find the true answer to the question
• However, research methods can be open to
• Systematic error bias
• Chance/random error imprecision
• Need to ensure that these errors are minimised in
  any study design

10

• BIAS

11
Bias (1)

• Any process at any stage of inference which
  tends to produce results of conclusions that
  differ systematically from the truth.

12
Bias (2)

• Unbiased

Biased
13
Bias (3)

• Many different types of bias over 100 sources
  of bias identified in analytical research.
• Principal sources of bias
• ?

14
Spot the potential for bias
15
Bias (4)

• Many different types of bias over 100 sources
  of bias identified in analytical research.
• Principal sources of bias
• selection
• performance
• attrition
• detection

16
Selection bias

• About
• who gets into the study and
• who gets which intervention

17
Who gets included in a study

• Bias may occur if subjects selected are likely to
  give systematically different answers to
  population of interest
• e.g. interview survey of attitudes to alcohol
  consumption in which interviewer stands outside
  pub.

18
Who gets which intervention

• In observational/non-randomised studies, bias may
  occur if there are systematic differences between
  patients receiving different interventions which
  influence outcome.
• In randomised trials, both known and unknown
  prognostic variables randomly distributed between
  the treatment groups unlikely to have
  systematic bias.
• TC Chalmers, P Celano, HS Sacks, and H Smith NEJM
  1983

19
Performance Bias

• Bias (at the intervention stage) may occur due
  to
• non specific placebo effects of intervention
• failure to blind patients to intervention group
• failure to provide adequate control activity

20
Attrition Bias

• In any research design attrition of sample may
  introduce bias
• More specifically
• In an RCT withdrawal, drop out or cross-over of
  individuals after assignment to treatment may
  introduce bias

21
Detection bias

• Failure to assess outcome blindly may introduce
  bias
• Research has shown that failure to allocate
  blindly was associated with reporting of a
  greater effect of intervention.
• Schultz KF, Chalmers I, Grimes DA Altman DG
  (1994) JAMA
• Schultz KF, Chalmers I, Hayes RJ Altman DG
  (1995) JAMA

22
Four main sources of bias
23
Minimising Bias

• Researchers need to
• be aware of potential biases when planning
  research, and
• develop methods to overcome most significant
  biases

24

• PRECISION

25
Precision - Background

• Research tends to use information from samples
  rather than involving the total population
• Sampling can lead to imprecise results (random
  variation around the truth)

26
PRECISION
Precise
Imprecise
27
PRECISION

• Precision is a function of sample size the
  larger the study, the more precise the
  information
• Researchers must, however, trade off sample size
  and resources available to undertake a study
• Statistical calculations are available to allow
  researchers to determine the most effective
  sample size to use (covered later in course)

28
Relationship between bias and precision
29
(No Transcript)
30

• GENERALISABILITY

31
Generalisability(1)

• The degree to which the results of a study hold
  true for situations other than those pertaining
  to the study, in particular for routine clinical
  practice.
• Deeks JJ, Glanville JM, Sheldon TA (1995).
  Undertaking systematic reviews of
• Research on effectiveness. NHS Centre for
  Reviews and Dissemination

32
Generalisability(2)

• Surveys often conducted in specific geographical
  areas of specific groups of individuals
• Important to consider how generalisable results
  are to other geographical settings and groups

33
Generalisability(3)
www.socialresearchmethods.net/kb/sampterm.htm
34
Generalisability in RCTs

• Often have tight inclusion criteria e.g. specific
  populations may be excluded
• Trials may be conducted in specific settings
  (e.g. secondary care settings)
• Important to consider whether study results are
  generalisable to patient groups and settings you
  are interested in

35
Generalisability(4)

• Linked to a studys
• external validity can we infer that any causal
  links identified in the study can apply to
  different types of persons, settings and times?

36
Estimating External Validity

• Construct validity relates to whether
  intervention designed or delivered in ways which
  would be unlikely to be replicated in other
  settings and which might modify effectiveness
• External validity/applicability relates to
  whether the study population was unrepresentative
  of the population to which we wish to generalise
  the findings

37

• APPLICABILITY

38
Applicability (1)

• Extension of generalisability from just exploring
  whether characteristics of study population are
  similar to those of population we wish to
  generalise to
• Increasing recognition that other factors may
  influence the effect of an intervention known
  as effect modifiers

39
Applicability(2) Potential effect modifiers

• Effect modifying factors which might influence
  applicability include
• patient factors (ethnicity, baseline risk)
• provider behaviour (learning curves in surgery)
• cultural factors (cultural beliefs in health
  promotion behaviour)

40
Applicability (3)

• Increasing interest in methods to identify which
  groups of patients are likely to benefit from
  treatment rather than average effect of
  treatment.
• Development of methods are likely to have
  significant impact upon design and conduct of
  research to improve applicability of study
  results.
• Glasziou and Irwig (1995). BMJ

41
Example of differential effects
Hypothetical drug trial to reduce seizure in
epilepsy
Favours control
Favours treatment
42
Some useful papers

• Schulz KF, Chalmers I, Hayes RJ, Altman DG.
  Empirical evidence of bias dimensions of
  methodological quality associated with estimates
  of treatment effects in controlled trials. JAMA
  1995 273408-12
• Jadad AR, Moore RA, Carroll D, et al. Assessing
  the quality of reports of randomized clinical
  trials Is blinding necessary? Controlled Clin
  Trials 1996 171-12.
• Moher D, Jadad AR, Tugwell P. Assessing the
  quality of randomized controlled trials current
  issues and future directions. Int J Tech Assess
  in Health Care 1996 12195-208.
• Clancy MJ, Overview of research Designs. Emerg
  Med J 20021946-549
• Brennan P, Croft P. Interpreting the results
  from observational research chance is not such a
  fine thing BMJ 1994309727-730

43
More

• Prevention of coronary heart disease with
  pravastatin in men with hypercholesterolemia. J
  Shepherd J, Cobbe SM, Ford I, Isles CG, Lorimer
  AR, Macfarlane PW, McKillop JH and CJ Packard CJ,
  for the West of Scotland Coronary Prevention
  Study Group. N.Engl.J.Med. 1995333(20)1301-1307.
  (has an example of a subgroup effect)
• Sackett DL. Bias in analytic research. J Chron
  Dis 1979325163. (hard to get a hold of this
  one)