Metaanalysis

1
Meta-analysis

• Funded through the ESRCs Researcher Development
  Initiative

Sessions 1.2-1.3 Effect Size Calculation
Department of Education, University of Oxford
2
Sessions 1.2-1.3 Effect Size Calculation
2
3
Effect size calculation

• The effect size makes meta-analysis possible
• It is based on the dependent variable (i.e.,
  the outcome)
• It standardizes findings across studies such that
  they can be directly compared
• Any standardized index can be an effect size
  (e.g., standardized mean difference, correlation
  coefficient, odds-ratio), but must
• be comparable across studies (standardization)
• represent magnitude direction of the
  relationship
• be independent of sample size
• Different studies in same meta-analysis can be
  based on different statistics, but have to
  transform each to a standardized effect size that
  is comparable across different studies

4
Sample size, significance and d effect size
XLS
5
Sample size, significance and d effect size
XLS
5
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
5
6
Simulate ds on homemade calculator (ES.xls)

• Change direction of effects
• Change Ns (equal or same?)
• Change SDs

XLS
6
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
6
7
Effect size as proportion in the Treatment group
doing better than the average Control group
person
79 of T above
69 of T above
57 of T above
7
8
Effect size as proportion of success in the
Treatment versus Control group (Binomial Effect
Size Display BESD)
Success 55 of T, 45 of C
Success 62 of T, 38 of C
Success 68 of T, 32 of C
8
9
Why effect size?

• Long focus on significance level (safe-guarding
  against Type I (a) error) today focus on
  practical and meaningful significance.
• Cohen, J. (1994). The earth is round (p lt .05),
  American Psychologist, 49, 9971003.

9
10
A short history of the effect size (Huberty,
2002 see also Olejnik Algina, 2000 for review
of effect sizes)
10
11
Power and effect size

• Power Finding what is out there
• Type II (b) error not finding what is out there
• Power (1 b) the probability of rejecting a
  false H0 hypothesis
• Power of .80 or .90 in primary research

11
12
Power, sought effect size, at significance level
a .05 in primary research (prior to conducting
study)
12
13
How meaningful is a small effect size?

• A small effect size changed the course of an RCT
  in 1987 placebo group participants were given
  aspirin instead (see Rosenthal, 1994, p. 242)

XLS
13
14
Effect sizes

• Within the one meta-analysis, can include studies
  based on any combination of statistical analysis
  (e.g., t-tests, ANOVA, correlation, odds-ratio,
  chi-square, etc).
• The art of meta-analysis is how to compute
  effect sizes based on non-standard designs and
  studies that do not supply complete data (see
  LipseyWilson_AppB.pdf).
• Convert all effect sizes into a common metric
  based on the natural metric given research in
  the area. E.g. d, r, OR

14
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
15
Effect size calculation

• Standardized mean difference
• Group contrast research
• Treatment groups
• Naturally occurring groups
• Inherently continuous construct
• Correlation coefficient
• Association between inherently continuous
  constructs
• Odds-ratio
• Group contrast research
• Treatment or naturally occurring groups
• Inherently dichotomous construct
• Regression coefficients and other multivariate
  effects
• Requires access to covariance-variance
  (correlation) matrices for each included study

15
16
Calculating ds (1)
Means and standard deviations
Almost all test statistics can be transformed
into an standardized effect size d
Correlations
d
P-values
F-statistics
t-statistics
other test statistics
16
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
16
17
Calculating ds (1)

• Represents a standardized group contrast on an
  inherently continuous measure
• Uses the pooled standard deviation
• Commonly called d

ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
18
Various contrast effect sizes

• Cohens d
• Hedges g
• Glasss D

18
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
19
Calculating d (1) using Ms, SDs and ns
Remember to code treatment effect in positive
direction!
19
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
20
ES_calculator.xls
20
20
21
Calculating d (2) using ES calculator, using Ms,
ns, and t-value
21
22
Calculating d (3) using ES calculator, using ns,
and t-value

• The treatment group scored higher than the
  control group at Time 2 (t28 4.11 plt.001).
• From sample description we learn that n1 n2

22
22
23
Calculating d (3) correcting for small sample bias

• Hedges proposed a correction for small sample
  size bias (ns lt 20)
• Must be applied before analysis

23
24
Calculating d (4) using ES calculator, using ns,
and F-value
Remember in a two-group ANOVA F t2
24
24
25
Calculating d (5) using ES calculator, using
p-value
The mean-level comparison was not significant (p
.53)
25
25
26
T-test table df (n1 ns 2)
Sometimes authors only report e.g., plt.01 (n
22). If so, use a conservative approach to
reading the t-test table.
NOTE When p n.s. some researchers code d 0
in data base
26
26
27
Example dataset so far (1) (ES_enter.sav)
27
27
28
Use all available tools for calculating the
following 5 effect sizes

• ES 6 MT 21, MC 20, nT 60, nC 60, t .55
• ES 7 MT 103.5, MC 100, SDT 22.0, SDC
  18.5, nT 45, nC 35,
• ES 8 nT 45, nC 40, p lt.05
• ES 9 nT 100, nC 120, F 8.73
• ES 10 nT 200, nC 160, t 5.66
• (see electronic document Correct ds for 5
  effect sizes.doc)

28
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
29
Example dataset so far (2) (ES_enter.sav)
29
29
30
Calculating d (11) using ES calculator, using
number of successful outcomes per group
30
30
31
Calculating d (11) using ES calculator, using
number of successful outcomes per group
31
31
32
Calculating d (12) using ES calculator, using
proportion of successes per group (53 vs. 48.5)
32
32
33
Calculating d (13) using paired t-test (only one
experimental group each person their own
control)
Dont use the SD of the change score!
r correlation between Time 1 and Time 2
33
33
34

• Calculating d (14) using paired t-test (only one
  experimental group)
• n (pairs) 90, t-value 6.5, r .70

34
34
35
Calculating d (15)

• The 20 participants increased .84 z-scores
  between time 1 and time 2 (plt.01)
• ES .84
• Correct for small sample bias

35
35
36
Example dataset so far 3 (ES_enter.sav)
Method difference mean contrast and gain scores
36
36
37
Summary of equations from Lipsey Wilson (2001)
(for more formulae see Lipsey Wilson Appendix B)
37
37
38
Weighting for mean-level differences

• The effect sizes are weighted by the inverse of
  the variance to give more weight to effects based
  on larger sample sizes
• Variance for mean level comparison is calculated
  as
• The standard error of each effect size is given
  by the square root of the sampling variance
• SE ? vi

38
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
38
39

• Enter_w.xls

39
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
39
40
Weighting for gain scores
T1 and T2 scores are dependent so we need to get
correlation between T1 and T2 into equation (not
always reported)

• SE for gain scores
• Inverse variance for gain scores

40
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
40
41

• Enter_w.xls

XLS
41
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
41
42
Compute the weighted mean ES and s.e. of the ES
in SPSS (var_ofES.sps) (1)
42
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
42
43
Compute the weighted mean ES and s.e. of the ES
in SPSS (var_ofES.sps) (2)
43
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
43
44
Compute the weighted mean ES and s.e. of the ES

• Weight the ES by the inverse of the s.e.
• The average ES
• Standard error of the ES

44
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
44
45

• Enter_w.xls

45
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
45
46
46
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
46
47
Funnel plot for x sample size, y ES

• Does average of ES converge toward the average of
  the largest (n) study?

95 C.I. 1.96 s.e. 99 C.I. 2.58
s.e. 99.9 C.I. 3.29 s.e.
47
48
Funnel plot including s.e. of ES

• ES in smaller sample has larger standard error
  (s.e.)

48
49
Population and sample
Sample
n size m mean d effect size
49
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
49
50
Calculating rs
Means and standard deviations (d)
Almost all test statistics can be transformed
into an standardized effect size r
c2 f
r
P-values
F-statistics
t-statistics
other test statistics
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
50
51
Correlations / relationships between variables

• rxy Pearsons product moment coefficient
  (continuous ? continuous)
• Rpb Bi-serial correlation (dichotomous ?
  continuous)
• c2 (dichotomous ? dichotomous)
• rsSpearmans rank-order coefficient (ordinal ?
  ordinal)
• And others, e.g.,
• f coefficient, Odds-Ratio (OR)
• Cramers V, Contingency coefficient C
• Tetrachoric and polychoric correlations . (etc)

51
52
Bias when dichotomising continuous variables

• X or Y are both truly continuous, but in the
  study either is dichotomised
• X continuous, Y 50/50 split gives an rpb that
  is 80 of its value, had it been continuous
• X or Y are both truly continuous, but both are
  dichotomised
• Maximum value of f if x 30/70 split and Y
  50/50 split is f .33

52
53
Calculating rs from d (1)
r can be used in all situations d can, but d
cannot be used in all situations where r is
appropriate
53
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
53
54
Calculating rpb (2)

• If inherently continuous X and Y, mean-contrast
  is a better option than rpb

54
55
Calculating r (3) from t-value

• Appropriate for both independent and dependent
  samples t-test values

Calculating r (4) from c2-value
55
55
56
Sources of error

• Cf. Structural Equation Model (circle latent/
  unobserved construct, rectangle manifest/
  observed variable)

rxy
Latent (unobserved) X
Latent (unobserved) Y
ryy
rxx
Manifest (observed) variable x
Manifest (observed) variable y
rxy
56
57
Alternatively transform rs into Fishers
Zr-transformed rs, which are more normally
distributed
57
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
57
58

• rr.xls

58
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
59
59
60
Calculating OR (chi2.sps)
60
61
61
62
62
63
Pearsons 5 studies escaping Enteric Fever (1904)
63
64
64
65
Effect size calculation
XLS
65
ESRC RDI One Day Meta-analysis workshop (Marsh,
OMara, Malmberg)
66
Steps in a meta-analysis
66
67
Structuring a database
67
68
Constructing a database
68
69
Organising effect sizes within study (1) Flat
dataset
69
70
Organising effect sizes within study (2)
hierarchical dataset (effect sizes nested
within study)
70
71
Organising effect sizes within study (3)
hierarchical dataset, with one construct per DV
per study
71
72
Organising effect sizes within study (4)
hierarchical dataset, with one DV per study
NOTE alternative to aggregating ESs within
study multilevel meta-analysis
72
73
Exercise effect size calculation (4
method/result extracts from journals)

• Do boys have higher general (global) self-concept
  (self-worth) than girls?
• Decide which effect size to use (d, r, OR)?
• Calculate appropriate effect sizes

73
74
Effect size literature

• Cohen, J. (1969). Statistical Power Analysis for
  the Behavioral Sciences, 1st Edition, Lawrence
  Erlbaum Associates, Hillsdale (2nd Edition,
  1988).
• Cohen, J. (1994). The earth is round (p lt .05),
  American Psychologist, 49, 9971003.
• Gwet, K. (2001). Handbook of interrater
  reliability. How to estimate the level of
  agreement between two of multiple raters.
  Gaithersburg STATAXIS Publishing.
• Huberty, C. J. (2002). A history of effect size
  indices. Educational and Psychological
  Measurement, 62, 227-240.
• McCartney, K., Rosenthal, R. (2000). Effect
  size, practical importance, and social policy for
  children. Child Development, 71, 173-180.
• Olejnik, S., Algina, J. (2000). Measures of
  effect size for comparative studies
  Applications, interpretations, and limitations.
  Contemporary Educational Psychology, 25, 241-286.

74