Sensitivity%20Analysis

1
Sensitivity Analysis
Meta-analysis in R with Metafor

• Examining outcomes under different decisions

2
Quality of Studies

• We make inclusion decisions and worry about the
  quality of the studies included in the analysis.
• Both inclusion decisions and study quality can be
  treated as a moderator if you have the data
• moderators were covered in an earlier video.

3
Models and Data

• Parameter estimates can be sensitive to
  influential data points
• Can be deviant in the sense of extreme distance
  from the mean or the regression line
• Can be very large sample
• Missing studies (availability bias)
• Outlier detection and removing studies
• Leave-one-out

4
Sensitivity Analyses

• Forest plot by precision (trim-and-fill covered
  in funnel plot video)
• Eggers regression (statistical test of
  asymmetry)
• Residuals
• Mean only
• Moderator in model
• Leave-one-out
• Statistics (Mean, I-squared, etc.)
• Graph (plot the mean and CI excluding each ES)

5
Forest Sleep AD
Note how the small studies are mostly all on the
bottom right. This suggests availability bias.
6
Eggers Regression
Ti effect size visampling variance of ES
Should be flat (centered) if no bias (beta1 is
zero). This shows small studies have higher
values. Significant negative slope (beta1) is
concerning.
Source Sutton (2009). In Cooper, Hedges,
Valentine (Eds) Handbook fo Research Synthesis
Methods p. 441
7
Funnel Asymmetry Test
Metafor offers several different tests for funnel
plot asymmetry. The regtest results are shown
below for the sleep data shown previously.
8
Finding Outliers
This is the result of an overall meta-analysis
with no moderators it is the overall result for
the varying (random) effects analysis.
Equivalent to a regression with intercept only.
9
Finding Outliers (3)
This analysis considers the study sample size and
the REVC in addition to the raw distance to the
mean. Look for large values the z refers to
the unit normal, so 2 is large, but I would
probably start with 2.5 or 3. Your call though.
I would sort by z (there are 92 residuals in this
analysis).
LMX data
10
Outliers 4
Here I have inserted a categorical independent
variable into the model. The residuals are now
studentized, deleted residuals that remain after
accounting for differences in culture (vertical
integration and collective orientation). As you
can see, adding the moderator made very little
difference for observation 12 (-2.08 to -2.05).
For an important moderator, there could be a
large difference.
11
Removing Observations
Remove one outlier negative sign to take out, no
negative sign, include only those.
Remove 2 outliers, run additional analyses
12
If you show that the impact of outliers and
nuisance variables (and moderators) is minimal or
at least not a threat to your inferences, then
your conclusions will be more credible.
13
Leave One Out

• In this sensitivity analysis, every study is
  removed, 1 by 1, and the overall ES is
  re-estimated.
• Helpful to judge the impact of influential
  studies (sometimes outlying ES, sometimes very
  large N)
• Can be run in rma if the model has no moderators
  (i.e., for overall ES or a subset of studies)
• Two uses for this
• Find and remove problematic studies
• Share summary of findings with your reader

14
McNatt Example

• R-code for McNatt