Metaanalysis

1
Meta-analysis

• Funded through the ESRCs Researcher Development
  Initiative

Session 1.3 Equations
Department of Education, University of Oxford
2
Steps in a meta-analysis
Session 1.3 Equations
3
Assumptions
3
4
Fixed effects
In this and following formulae, we will use the
symbols d and d to refer to any measure for the
observed and the true effect size, which is not
necessarily the standardized mean difference.

• Formula for the observed effect size in a fixed
  effects model
• Where
• dj is the observed effect size in study j
• d is the true population effect
• and ej is the residual due to sampling variance
  in study j

5
Calculating the observed effect size

• To calculate the overall mean observed effect
  size (dj in the fixed effects equation)
• where wi weight for the individual effect size,
  and di the individual effect size.

6
Weighting

• The effect sizes are weighted by the inverse of
  the variance to give more weight to effects based
  on large sample sizes
• The standard error of each effect size is given
  by the square root of the sampling variance
• SE ? vi
• The variances are calculated differently for each
  type of effect size.

6
7
Weighting

• Variance for standardised mean difference effect
  size is calculated as
• Where n1 sample size of group 1, n2 is the
  sample size of group 2, and di the effect size
  for study i.
• Variance for correlation effect size is
  calculated as
• Where ni is the total sample size of the study

7
8
Predictors in fixed effects model

• Expand the general model to include predictors
• Where
• ßs is the regression coefficient (regression
  slope) for the explanatory variable.
• Xsj is the study characteristic (s) of study j.

9
Example of fixed effects model with predictor
Example Gender as a predictor of achievement
10
Random effects

• Formula for the observed effect size in a random
  effects model
• Where
• dj is the observed effect size in study j
• d is the mean true population effect size
• uj is the deviation of the true study effect size
  from the mean true effect size
• and ej is the residual due to sampling variance
  in study j

11
Calculating the observed effect size

• To calculate the overall mean observed effect
  size (dj in the random effects equation)
• where wi weight for the individual effect size,
  and di the individual effect size.

12
Weighting in random effects models

• Random effects differs from fixed effects in the
  calculation of the weighting (wi)
• The weight includes 2 variance components
  within-study variance (vi) and between-study
  variance (v?)
• The new weighting for the random effects model
  (wiRE) is given by the formula
• vi is calculated the same as in the fixed effects
  models.

12
13
Weighting in random effects models

• v? is calculated using the following formula
• Where Q Q-statistic (measure of whether effect
  sizes all come from the same population)
• k number of studies included in sample
• wi effect size weight, calculated based on
  fixed effects models.

13
14
Weighting in random effects models

• Thus, larger studies receive proportionally less
  weight in RE model than in FE model.
• This is because a constant is added to the
  denominator, so the relative effect of sample
  size will be smaller in RE model

14
15
Random effects models

• If the homogeneity test is rejected (it almost
  always will be), it suggests that there are
  larger differences than can be explained by
  chance variation (at the individual participant
  level). There is more than one population in
  the set of different studies.
• The random effects model determines how much of
  this between-study variation can be explained by
  study characteristics that we have coded.

16
Predictors in random effects model

• Expand the general model to include predictors
• Where
• ßs is the regression coefficient (regression
  slope) for the explanatory variable.
• Xsj is the study characteristic (s) of study j.

17
Example of random effects model with predictor
Example Gender as a predictor of achievement
18
Multilevel modelling

• Formula for the observed effect size in a
  multilevel model
• Where
• dj is the observed effect size in study j
• ?0 is the mean true population effect size
• uj is the deviation of the true study effect size
  from the mean true effect size
• and ej is the residual due to sampling variance
  in study j
• Note This model treats the moderator effects as
  fixed and the ujs as random effects.

19
Predictors in a multilevel model

• In this equation, predictors are included in the
  model.
• ?s is the regression coefficient (regression
  slope) for the explanatory variable. (Equivalent
  to ß in multiple regression.)
• Xsj is the study characteristic (s) of study j.

20
Example of multilevel model with predictor
Example Gender as a predictor of achievement
21
Simplifying the multilevel equation

• If between-study variance 0, the multilevel
  model simplifies to the fixed effects regression
  model
• If no predictors are included the model
  simplifies to random effects model
• If the level 2 variance 0 , the model
  simplifies to the fixed effects model

22
In practice...

• Many meta-analysts use an adaptive (or
  conditional) approach
• IF between-study variance is found in the
  homogeneity test
• THEN use random effects model
• OTHERWISE use fixed effects model

23
In practice...

• Fixed effects models are very common, even though
  the assumption of homogeneity is implausible
  (Noortgate Onghena, 2003)
• There is a considerable lag in the uptake of new
  methods by applied meta-analysts
• Meta-analysts need to stay on top of these
  developments by
• Attending courses
• Wide reading across disciplines

24
What do the models involve?
24
25
Conducting fixed effects meta-analysis

• Usually start with a Q-test to determine the
  overall mean effect size and the homogeneity of
  the effect sizes (MeanES.sps macro)
• If there is significant homogeneity, then
• 1) should probably conduct random effects
  analyses instead
• 2) model moderators of the effect sizes
  (determine the source/s of variance)

26
Q-test of the homogeneity of variance
The homogeneity (Q) test asks whether the
different effect sizes are likely to have all
come from the same population (an assumption of
the fixed effects model). Are the differences
among the effect sizes no bigger than might be
expected by chance?
di effect size for each study (i 1 to k)
mean effect size a weight for each study
based on the sample size However, this
(chi-square) test is heavily dependent on sample
size. It is almost always significant unless the
numbers (studies and people in each study) are
VERY small. This means that the fixed effect
model will almost always be rejected in favour of
a random effects model.

27
Fixed effects mean effect size
27
28
ANOVA

• The analogue to the ANOVA homogeneity analysis is
  appropriate for categorical variables
• Looks for systematic differences between groups
  of responses within a variable
• Easy to implement using MetaF.sps macro
• MetaF ES d /W Weight /GROUP TXTYPE /MODEL
  FE.

29
Multiple regression

• Multiple regression homogeneity analysis is more
  appropriate for continuous variables and/or when
  there are multiple variables to be analysed
• Tests the ability of groups within each variable
  to predict the effect size
• Can include categorical variables in multiple
  regression as dummy variables
• Easy to implement using MetaReg.sps macro
• MetaReg ES d /W Weight /IVS IV1 IV2 /MODEL
  FE.

30
Conducting random effects meta-analysis

• Like the FE model, RE uses ANOVA and multiple
  regression to model potential moderators/predictor
  s of the effect sizes, if the Q-test reveals
  significant heterogeneity
• Easy to implement using MetaF.sps macro (ANOVA)
  or MetaReg.sps (multiple regression).
• MetaF ES d /W Weight /GROUP TXTYPE /MODEL
  ML.
• MetaReg ES d /W Weight /IVS IV1 IV2 /MODEL
  ML.

31
Random effects mean effect size
31
32
Conducting multilevel model analyses

• Similar to multiple regression, but corrects the
  standard errors for the nesting of the data
• Start with an intercept-only (no predictors)
  model, which incorporates both the outcome-level
  and the study-level components
• This tells us the overall mean effect size
• Is similar to a random effects model
• Then expand the model to include predictor
  variables, to explain systematic variance between
  the study effect sizes

32
33
Multilevel set-up

• (MLwiN screenshot)

34
Multilevel mean effect size

• Using the same simulated data set with n 15

35
Comparability of random and multilevel models (no
predictors)
36
Van den Noortgate Onghena (2003)

• The random effects is better than the fixed
  effects approach in almost all conceivable cases
• The results of the simulation study suggest that
  the maximum likelihood multilevel approach is in
  general superior to the fixed-effects approaches,
  unless only a small number of studies is
  available. For models without moderators, the
  results of the multilevel approach, however, are
  not substantially different from the results of
  the traditional random-effects approaches (p.
  765)

37
Conclusions

• Multilevel models
• build on the fixed and random effects models
• account for between-study variance (like random
  effects)
• Are similar to multiple regression, but correct
  the standard errors for the nesting of the data.
  Improved modelling of the nesting of levels
  within studies increases the accuracy of the
  estimation of standard errors on parameter
  estimates and the assessment of the significance
  of explanatory variables (Bateman and Jones,
  2003).
• Multilevel modelling is more precise when there
  is greater between-study heterogeneity
• Also allows flexibility in modelling the data
  when one has multiple moderator variables
  (Raudenbush Bryk, 2002)

38
Other (potential) benefits of MLM

• Multilevel modelling has the promise of being
  able to include multivariate data still being
  developed
• Easy to implement in MLwiN (once you know how!)
• See worked examples for HLM, MLwiN, SAS, Stata
  at
• http//www.ats.ucla.edu/stat/examples/ma_hox/defau
  lt.htm

39
References

• Lipsey, M. W., Wilson, D. B. (2001). Practical
  meta-analysis. Thousand Oaks, CA Sage
  Publications.
• Van den Noortgate, W., Onghena, P. (2003).
  Multilevel meta-analysis A comparison with
  traditional meta-analytical procedures.
  Educational and Psychological Measurement, 63,
  765-790.
• Wilsons meta-analysis stuff website
  http//mason.gmu.edu/dwilsonb/ma.html
• Raudenbush, S.W. and Bryk, A.S. (2002).
  Hierarchical Linear Models (2nd Ed.).Thousand
  Oaks Sage Publications.