Metaanalysis

1
Meta-analysis

• Funded through the ESRCs Researcher Development
  Initiative

Session 3.2 Multivariate meta-analysis
Department of Education, University of Oxford
2
Steps in a meta-analysis
Session 3.2 Multivariate meta-analysis
3
Multivariate datasets
4
What do we mean by "multivariate"?

• Involves the analysis of multiple outcomes
  simultaneously
• Multiple outcomes could be due to
• Different outcomes (e.g., math achievement and
  verbal achievement)
• Correlations with multiple variables (e.g., age
  with achievement and age with aspirations)
• Evaluation of different treatments in the same
  publication
• More than one control/comparison group

5
Why is this a problem?

• Violations of independence occur when studies
  produce multiple effect sizes due to the presence
  of multiple treatment groups or multiple outcome
  measures
• Effect sizes from the same study are more likely
  to have a higher correlations than effect sizes
  from different studies
• Issue of within versus between-study variation

6
Options for dealing with multiple outcomes

• Choose one outcome of interest
• Separate analyses on each outcome
• Averaging the effect sizes (one outcome study)
• Shifting unit of analysis (Cooper, 1998)
• Multivariate multilevel modelling

7
Choose one outcome of interest

• Select the outcome that is of most interest
• This is appropriate for many research questions
• However, does not allow contrasts between
  outcomes, thereby restricting the questions you
  can ask

8
Conduct separate analyses on each outcome or
treatment group

• For each analysis, only one outcome (effect size)
  per study is contributed to the analysis
• E.g., run separate analyses on maths achievement
  effect sizes, and a different set of analyses on
  the verbal achievement effect sizes
• The effect sizes are independent within the
  particular analysis, but does not allow direct
  comparison between the outcomes
• Therefore, this may not always make sense for the
  research question under consideration (Rosenthal
  Rubin, 1986)

9
Average the outcomes

• Establish an independent set of effect sizes by
  calculating the average of the effect sizes in
  the study
• E.g., achievement, intelligence, satisfaction,
  personality, obesity
• However, the dependent variables need to be
  almost perfectly correlated for this method to
  work, because the mean effect size gives an
  estimate that is lower than expected (Rosenthal
  Rubin, 1986)
• To make the results meaningful, outcomes should
  be conceptually similar

10
Shifting unit of analysis (Cooper, 1998)

• The outcomes are aggregated depending on the
  level of analysis of interestthe study or
  outcome level
• At the study level, all effect sizes from within
  a study are aggregated to produce one outcome per
  study
• For each moderator analysis, effect sizes are
  aggregated based upon the particular moderator
  variable, such that each study only includes one
  effect size per outcome on that particular
  variable

11
Shifting unit of analysis example

• The effect sizes for two self-concept domains
  (e.g., physical and academic self-concept) from
  the same primary study would initially be
  averaged to produce a single effect size for
  calculations involving the overall effect size
  for the sample (study level)
• For the moderator analyses, the two self-concept
  domains would be considered separately if the
  type of domain was of interest, but would be
  aggregated if the moderator variable of interest
  was, say, the type of control group
• This means that the n of effect sizes
  contributing to the analysis will change
  depending on the variables being examined

12
Shifting unit of analysis
13
Shifting unit of analysis - aggregate by
intervention
One effect size per study for maths
interventions, one per study for verbal
interventions
14
Shifting unit of analysis - aggregate by outcome
One math effect size per study, one verbal
effect size per study
15
Shifting unit of analysis

• Although this strategic compromise does not
  eliminate the problem of independence, this
  approach minimizes violations of assumptions
  about the independence of effect sizes, whilst
  preserving as much of the data as possible
  (Cooper, 1998)
• Probably the most popular way of dealing with
  multiple outcomes in fixed and random effects
  models when explicitly interested in comparing
  different outcomes

16
The multilevel approach

• Multilevel modelling accounts for dependencies in
  the data because its nested structure allows for
  correct estimation of standard errors on
  parameter estimates and therefore accurate
  assessment of the significance of predictor
  variables (Bateman Jones, 2003 Hox de Leeuw,
  2003 Raudenbush Bryk, 2002).

17
Multilevel modelling assumptions

• Meta-analytic data is inherently hierarchical
  (i.e., effect sizes nested within studies) and
  has random error that must be accounted for
• Effect sizes are not necessarily independent
• Allows for multiple effect sizes per study
• Also provides more precise and less biased
  estimates of between-study variance than
  traditional techniques

18
Example multilevel

• Scholastic Aptitude Test (SAT) coaching
  effectiveness data reported in Kalaian and
  Raudenbush (1996), and Kalaian Kasim (in press)
• The differences between the coached and uncoached
  groups on SAT scores in the collection of the SAT
  coaching effectiveness studies
• SAT tests are widely claimed to be so broad and
  generic (almost IQ-like) that could not be
  affected by short-term training program. Others
  suggest that a limited amount of
  "familiarisation" is useful but not much beyond
  this (i.e., non-linear effect of hours).

19
Meta-analytic data

• Meta-analytic data information
• Study ID
• Constant (cons)
• Effect-size for verbal SAT scores (dv)
• Effect-size for maths SAT scores (dm)
• Sampling variance (SE) and covariance (cov_VM) of
  the effect sizes
• Explanatory variables (study and sample
  characteristics) (hours, logHR, year)

20
Import data into MLwiN
21
(No Transcript)
22

• Click on "responses" at the bottom of the screen
• select "dv" and "dm" (the effect sizes for
  verbal and maths achievement, respectively)

23

• Click on the equation
• indicate a two level model with L2study, L1
  resp_indicator
• Click done button

24

• Click "add term"
• Select variable cons
• Click add separate coefficients button
• Click Estimates

2 3
1
4
25

• Right-click on cons.dm in the equation
• Select j
• Click on Done

• Right-click on cons.dv in the equation
• Select j
• Click on Done

26
Click on Estimates Your screen should look like
this
27

• Click "add term"
• Select variable SE_V
• Click add separate coefficients button

28

• Click on estimates to reveal numbers
• Right click on SE_V.dm
• Click on Delete Term

29

• Click "add term"
• Select variable SE_M
• Click add separate coefficients button

30

• Right click on SE_M.dv
• Click on Delete Term

31

• Right-click on SE_M.dm in the equation
• Select j , unselect Fixed parameter
• Click on Done

• Right-click on SE_V.dv in the equation
• Select j , unselect Fixed parameter
• Click on Done

32
Your model now looks like this. Some of the
parameters in the random part of the model (the
us) do not make sense to be estimated. The only
random parameters that we want are those on the
diagonal in the variance-covariance matrix.
33
You can delete the unnecessary random parameters
by clicking on them. For example, click on
?u02
The following screen will pop up. Click on Yes
34
Delete all of the off-diagonal random parameters
for the SEs, until your variance-covariance
matrix looks like this
35
Now we need to add the covariance value Click on
add term Then select the covariance term,
cov_VM, and click on add Common coefficient
36
Covariance term

• The covariance term needs to be manually
  calculated (see Kalaian Kasim, in press)
• The formula is
• Where n1 and n2 are the sample sizes for the 2
  groups
• Rip,ip is the correlation between the two
  outcomes
• dip and dip are the two effect size outcomes

37

• Click ß4cov_VM.12j
• Select the options as below

2
38
Your equation window should now look like this.
Delete the off-diagonal covariance components for
u4j by clicking on them
39
Your equation window should now look like this.
40
Under Model in the menu bar, click on
Constrain Parameters The following window
should pop up
Click on the random radio button
41
Attach random constraints

• Select the SE and cov_VM variances to be
  constrained by entering a 1 in the boxes
• Set them to equal 1
• Choose a free column to store the constraint
  matrix in. In this case, we used C20
• Click on attach random constraints
• Go back to the equations window

1
2
3
4
42
Select Estimation from the menu bar, then
RIGLS Click Done when the window pops up
43
Your model should look something like this...
(you may need to click on Estimates to show the
numbers in blue font) Click on START when you are
ready to run the model
44
(No Transcript)
45
Results

• On average, students scored higher on maths SAT
  than verbal SAT
• However, variance was larger for maths
• There was no significant between-study variation
  for maths (.012) or verbal (.004) SAT scores
• Given that there is no significant between-study
  variation, we would not normally fit the model
  with predictors.

46
From Kalaian Kasim (in press)
Figure 4. Box Plot of SAT-Verbal and SAT-Math
Effect Sizes
47
Adding a predictor

• Lets look at a predictor anyway for
  demonstration purposes!
• Test whether a coaching intervention improves
  maths and verbal SAT scores
• Will the effects (size, direction,
  significance) of the coaching be the same for the
  two outcomes?

48

• Add Term
• Select LogEHR (not LogHR)
• Click on grand mean (mean 19 hours)
• Click on add separate coefficients
• Run the model (start)

49
Studies with Log coaching hours gt2.75 (which is
the study with 15 non-logged, raw hours) will
have a very nearly significant positive effect on
Verbal SAT scores (ß .102). Studies with Log
coaching hours gt2.75 will have a significant
positive effect on Maths SAT scores (ß .290)
50
Caveat about multivariate approach

• Calculating the covariance (in this case,
  cov_VM) requires knowing the correlation
  between the outcomes
• Often, primary studies do not report the
  correlations between the outcomes. Some methods
  are being developed that bypass this problem
• Riley, Thompson, Abrams (2008) An alternative
  model for bivariate random-effects meta-analysis
  when the within-study correlations are unknown
• However, these are confined to bivariate studies
• What to do when more than 2 outcomes?
• Model will become very complex
• Currently under development

51
Take home message

• The multivariate results account for the
  covariance between the verbal SAT and Maths SAT
  effect sizes.

52
References

• Kalaian, S. A. Kasim, R. M. (in press).
  Applications of Multilevel Models for Meta-
  Analysis. Multilevel Analysis of Educational
  Data. OConnell, A. and McCoach, B. D. (Eds.).
  Information Age Publishing.
• Kalaian, H. A., Raudenbush, S. W. (1996). A
  Multivariate Mixed-Effects Linear Model for
  Meta-Analysis. Psychological Methods, 1(3).
  227-235.
• R. D. Riley, J. R. Thompson, K. R. Abrams
  (2008). An alternative model for bivariate
  random-effects meta-analysis when the
  within-study correlations are unknown,
  Biostatistics, 9, 172-186.