Heterogeneity

1
Heterogeneity

• Reza Yousefi Nooraie

2
What is heterogeneity?

• Variability in effect size estimates which
  exceeds that expected from sampling error alone.

3
Sources of Variation over Studies

• Inter-study variation may exist
• Sampling error may vary among studies (sample
  size)
• Characteristics may differ among studies
  (population, intervention)

4
Sources of Heterogeneity

• Study design (inclusion criteria, treatment,
  duration)
• Study quality (randomisation, blinding etc)
• Individual level (prognostic factors)
• Outcomes (chance results)

5
How to Identify Heterogeneity

• Common sense
• are the patients, interventions and outcomes in
  each of the included studies sufficiently similar
• Statistical tests

6
Statistical Tests of Homogeneity (heterogeneity)

• Homogeneity calculations
• Ho studies are homogeneous
• Based on testing the sum of weighted differences
  between the summary effect and individual effects
• Calculate Mantel Haenszel Q, where
• Q ?weighti x (lnORmh - lnORi)2
• If p lt 0.05, then there is significant
  heterogeneity.

7
Statistical Tests of Homogeneity (heterogeneity)

• Power of such statistical tests is low
• (a non-significant test does not rule out
  clinically important heterogeneity)

8

• A useful statistic for quantifying inconsistency
  is
• I2 (Q df)/Q ? 100
• where Q is the chi-squared statistic and df is
  its degrees of freedom

9

• This describes the percentage of the variability
  in effect estimates that is due to heterogeneity
  rather than sampling error (chance).
• A value greater than 50 may be considered
  substantial heterogeneity.

10
How to deal with Heterogeneity

• If homogenous, use fixed effects model
• random will give same results
• fixed is computationally simpler
• If heterogeneousthen first ask why?!
• In the face of heterogeneity, focus of analysis
  should be to describe possible sources of
  variability
• attempt to identify sources of important subgroup
  differences
• .

11
How to deal with Heterogeneity

• 1. Do not pool at all
• 2. Ignore heterogeneity use fixed effects model
• difficult to interpret estimate
• 3. Explore heterogeneity
• subgroup analysis
• meta-regression
• 4. Random effects model

12
Methodologic choices in heterogeneous data
13

• A systematic review need not contain any
  meta-analyses.
• If there is considerable variation in results, it
  may be misleading to quote an average value

14
Exploring Heterogeneity
15
Exploring Heterogeneity
16
Fixed effects model

• All trials are measuring a single, true effect
• The reason for any difference between the effect
  in an individual trial and this true effect is
  chance

17
Fixed effects model
18
Fixed effects model

• consider only within-study variability.
• assumption is that studies use identical methods,
  patients, and measurements that they should
  produce identical results - any differences are
  only due to within-study variation only.
• Answer the question
• Did the treatment produce benefit on average in
  the studies at hand?

19
Fixed Effects Model

• Combine these using a weighted average
• pooled estimate
• where weight 1 / variance of estimate
• Assumes a common underlying effect behind every
  trial

sum of (estimate ? weight) sum of weights
20
Fixed-Effects Model
Study Measure Std Error Weight 1 Y1 s1 W1 2 Y
2 s2 W2 . . . . . . . . . . . . k Yk sk
Wk (no association Yi0)
Overall Measure
21
Random Effects models

• Each trial is measuring a different, true effect
• The true effects for each trial are normally
  distributed
• There is a true average effect
• The reason for any difference between the effect
  in an individual trial and this average effect is
  both the difference between the true effect for
  the trial and this average, and chance.

22
Random Effects models
23
Random Effects models

• consider both between-study and within-study
  variability.
• assumption is that studies are a random sample
  from the universe of all possible studies.
• Answer the question
• Will the treatment produce a benefit on
  average?
• Note that random effects models do not adjust
  for, account for, or explain heterogeneity
• A random effects model does not therefore solve
  the problem of heterogeneity!

24
Random-Effects Model

• Two sources of variation
• within studies (between patients)
• between studies (heterogeneity)
• Weight
• When heterogeneity exists we get
• a different pooled estimate
• a wider confidence interval
• a larger p-value

1 Variance heterogeneity
25
Random Effects Model
If is known then MLE of is
26

• The random effects estimate and its confidence
  interval address the question
• what is the average treatment effect?
• while the fixed effect estimate and its
  confidence interval addresses the question
• what is the best estimate of the treatment
  effect?

27
Fixed Effects
Random Effects
28
Fixed Effects
Random Effects
29
Random effects models

• DerSimonian and Laird statistic
• Uses odds ratios only!
• lnORdl ?(wi x lnORi) / ?wi
• wi 1 / D (1/wi)
• wi 1 / variancei
• D (Q - (S - 1) x ?wi ) / (?wi)2 - ?wi2
• Q ?wi x (lnORi - lnORmh)2
• CI exp(lnORdl 1.96 x (variances)0.5
• variances ?weighti

30
Use of the Random Effects Model?.

• Many observers dispute the rationale for
  random-effect based analyses.

31
Effect of model choice on study weights
Larger studies receive proportionally less weight
in RE model than in FE model
32

• This is because small studies are more
  informative for learning about the distribution
  of effects across studies than for learning about
  an assumed common treatment effect.
• Care must be taken that random effects analyses
  are applied only when the idea of a random
  distribution of treatment effects can be
  justified.
• In particular, if results of smaller studies are
  systematically different from results of larger
  ones, which can happen as a result of publication
  bias or low study quality bias, then a random
  effects meta-analysis will exacerbate the effects
  of the bias.
• In this situation it may be wise to present
  neither type of meta-analysis, or to perform a
  sensitivity analysis in which small studies are
  excluded.

33

• When there are few trials or the trials are
  small, a random effects analysis will provide
  poor estimates of the width of the distribution
  of treatment effects.
• The Mantel-Haenszel method will provide more
  robust estimates of the average treatment
  effect(at the cost of ignoring the observed
  heterogeneity.)

34

• Thus,
• Random-effect model is more susceptible to
  publication bias.