Using metaanalyses in your literature review

1
Using meta-analyses in your literature review

• BERA Doctoral Workshop
• 3rd September 2008
• Professor Steven Higgins
• Durham University
• s.e.higgins_at_durham.ac.uk

2
Aims

• To support understanding of meta-analysis of
  intervention research findings in education
• To extend understanding of reviewing quantitative
  research literature
• To describe the techniques and principles of
  meta-analysis involved to support understanding
  of its benefits and limitations
• To provide references and examples to support
  further work.

3
ESRC Researcher Development Initiative

• Quantitative synthesis of intervention research
  findings in education
• Collaboration between
• Durham University
• York University
• Institute of Education, London

4
Why review?

• Ask the person next to you what the purpose of
  the literature review is in their thesis
• See how many different purposes you can think of
• Join another pair and identify which are the 3
  you think are the most important

5
Why review?

• Summarise existing knowledge
• What we know, and how we know it
• For what purpose?
• Expectation
• Scenery
• State of the art (summary)
• Positioning (conceptual)
• Progressing knowledge (logic)

6
The PhD literature review

• Narrative summary of the area
• Grand tour of the concepts and terminology
• Synthesis of empirical findings
• Background to the study

7
A systematic review

• is usually more comprehensive
• is normally less biased, being the work of more
  than one reviewer
• is transparent and replicable
• (Andrews, 2005)

8
Examples of systematic reviews

• EPPI Centre
• UK based - wide range of educational topics
• The Campbell Collaboration
• 5 education reviews
• Best Evidence Encyclopedia
• Johns Hopkins - aimed at practice

9
Systematic reviewing

• Key question
• Search protocol
• Inclusion/exclusion criteria
• Coding and Mapping
• In-depth review (sub-question)
• Techniques for systematic synthesis

10
Systematic reviews

• Research and policy
• Specific reviews to answer particular questions
• What works? - impact and effectiveness research
  with a tendency to focus on quantitative and
  experimental designs

11
Literature reviewing - conceptual relations
Narrative review
Systematic review
Meta-analysis
12
Meta-analysis

• Synthesis of quantitative data
• Cumulative
• Comparative
• Correlational
• Surveys educational research (Lipsey and
  Wilson, 2001)

13
Origins

• 1952 Hans J. Eysenck concluded that there were
  no favorable effects of psychotherapy, starting a
  raging debate which 25 years of evaluation
  research and hundreds of studies failed to
  resolve
• 1978 To proved Eysenck wrong, Gene V. Glass
  statistically aggregated the findings of 375
  psychotherapy outcome studies
• Glass (and colleague Smith) concluded that
  psychotherapy did indeed work - the typical
  therapy trial raised the treatment group to a
  level about two-thirds of a standard deviation on
  average above untreated controls the average
  person received therapy finished the experiment
  in a position that exceeded the 75th percentile
  in the control group on whatever outcome measure
  happened to be taken (Glass, 2000). Glass called
  the method meta-analysis
• ( adapted from Lipsey Wilson, 2001)

14
Historical background

• Underpinning ideas can be identified earlier
• K. Pearson (1904)
• Averaged correlations for typhoid mortality after
  inoculation across 5 samples
• R. A. Fisher (1944)
• When a number of quite independent tests of
  significance have been made although few or
  none can be claimed individually as significant,
  yet the aggregate gives an impression that the
  probabilities are on the whole lower than would
  often have been obtained by chance (p. 99).
• Source of the idea of cumulating probability
  values
• W. G. Cochran (1953)
• Discusses a method of averaging means across
  independent studies
• Set out much of the statistical foundation for
  meta-analysis (e.g., Inverse variance weighting
  and homogeneity testing)
• ( adapted from Lipsey Wilson, 2001)

15
Significance versus effect size

• Traditional test is of statistical significance
• The difference is unlikely to have occurred by
  chance
• However it may not be
• Large
• Important, or even
• Educationally significant

16
The rationale for using effect sizes

• Traditional reviews focus on statistical
  significance testing
• Highly dependent on sample size
• Null finding does not carry the same weight as
  a significant finding
• Meta-analysis focuses on the direction and
  magnitude of the effects across studies
• From Is there a difference? to How big is the
  difference?
• Direction and magnitude represented by effect
  size

17
Effect size

• Comparison of impact
• Same AND different measures
• Significance vs effect size
• Does it work? vs How well does it work?

18
Effect size

• Standardised way of looking at gain scores
• Different methods for calculation
• Experimental group mean - Control mean/ Standard
  deviation

19
What is effect size?

• Standardised way of looking at difference
• Different methods for calculation
• Odds Ratio
• Correlational (Pearsons r)
• Standardised mean difference
• Difference between control and intervention group
  as proportion of the dispersion of scores

20
Calculating effect size

• Control group gain minus experimental group gain
  divided by the standard deviation of the groups

21
Effect size and impact
22
Interpreting effect sizes

• Relative effects - average is about 0.37 - 0.4
  (Sipe and Curlette, 1997 Hattie, Biggs and
  Purdie, 1996)
• Doing something different makes a difference
• Visualising the difference

23
How much is the impact?

• 0.1 percentile gain of 6 points
• ie a class ranked 50th in a league table of 100
  schools would move from 50th to about 44th place
• 0.5 percentile gain of 20 points
• ie move from 50th to 30th place
• 1.0 percentile gain of 34 points
• ie move from 50th to 16th place

24
Other interpretations

• 0.2 small difference in height between 15-16
  year olds
• 0.5 medium difference in height between 14
  and 18 year olds
• 0.8 large difference in height between 13 and
  18 year olds

Cohen 1969
25
Meta-analysis

• Key question
• Search protocol
• Inclusion/exclusion criteria
• Coding
• Statistical exploration of findings
• Mean
• Distribution
• Sources of variance

26
(No Transcript)
27
Some findings from meta-analysis

• Pearson et al. 2005
• 20 research articles, 89 effects related to
  digital tools and learning environments to
  enhance literacy acquisition. Weighted effect
  size of 0.489 indicating technology can have a
  positive impact on reading comprehension
• Bernard et al. 2004
• Distance education and classroom instruction -
  232 studies, 688 effects - wide range of effects
  (heterogeneity) asynchronous DE more effective
  than synchronous

28
More findings

• Hattie and Timperley, 2007
• The Power of Feedback, synthesis of other
  meta-analyses on feedback to provide a conceptual
  review 196 studies, 6972 effects - average effect
  of feedback on learning 0.79

29
Rank (or guess) some effect sizes

• Formative assessment
• CASE (Cognitive Acceleration Through Science
  Education)
• Individualised instruction
• ICT
• Homework
• Direct instruction

30
Rank order of effect sizes

• 1. 04 CASE (Cognitive Acceleration Through
  Science Education) (Boys science GCSE - Adey
  Shayer, 1991)
• 0.6 Direct instruction (Sipe Curlette, 1997)
• 0.43 Homework (Hattie, 1999)
• 0.32 Formative assessment (KMOFAP)
• 0.31 ICT (Hattie, 1999)
• 0.1 Individualised instruction (Hattie, 1999)

31
Super-syntheses

• Syntheses of meta-analyses
• Relative effects of different interventions
• Assumes variation evens out across studies with a
  large enough dataset (Marzano/Hattie) or attempts
  to control for the variation statistically (Sipe
  Curlette)

32
Hattie Biggs and Purdie, 1996

• Synthesis of study skills interventions
• Meta-analysis of 51 studies of study skills
  interventions. Categorised the inverventions
  using the SOLO model (Biggs Collis, 1982),
  classified studies into four hierarchical levels
  of structural complexity and as either near or
  far transfer. The results support situated
  cognition, and that training for other than
  simple mnemonic tasks should be in context, use
  tasks within the same domain as the target
  content, and promote a high degree of learner
  activity and metacognitive awareness.
• (average effect 0.4)

33
Sipe and Curlette, 1997

• A metasynthesis of factors relating to
  educational achievement - testing Walbergs
  educational productivity model - synthesis of
  103 meta-analyses

34
Marzano, 1998

• Theory driven
• Self system - metacognition - cognition/
  knowledge
• Self - 0.74
• Metacogntive 0.72
• Cognitive 0.55

35
Discussion

• Work with a colleague to put the statements in
  order of how comparable you think the research
  findings are
• Join another pair (or pairs) and decide how
  comfortable would you be with comparing the
  findings

36
Issues and challenges in meta-analysis

• Conceptual
• Reductionist - the answer is 42
• Comparability - apples and oranges
• Atheoretical - flat-earth
• Technical
• Heterogeneity
• Publication bias
• Methodological quality

37
Reductionist or flat earth critique

• The flat earth criticism is based on Lee
  Cronbachs assertion that a meta-analysis looks
  at the big picture and provides only a crude
  average. According to Cronbach,
• some of our colleagues are beginning to sound
  like a Flat Earth Society. They tell us that the
  world is essentially simple most social
  phenomena are adequately described by linear
  relations one-parameter scaling can discover
  coherent variables independent of culture and
  population and inconsistencies among studies of
  the same kind will vanish if we but amalgamate a
  sufficient number of studiesThe Flat Earth folk
  seek to bury any complex hypothesis with an
  empirical bulldozer (Cronbach, 1982, in Glass,
  2000).

38
Comparability

• Apples and oranges
• Same test
• Different measures of the same construct
• Different measures of different constructs
• What question are you trying to answer?
• How strong is the evidence for this?

Of course it mixes apples and oranges in the
study of fruit, nothing else is sensible
comparing apples and oranges is the only endeavor
worthy of true scientists comparing apples to
apples is trivial (Glass, 2000).
39
Empirical not theoretical?

• What is your starting point?
• Conceptual/ theoretical critique
• Marzano
• Hattie
• Sipe and Curlette

40
Technical issues

• Interventions
• Publication bias
• Methodological quality
• Sample size
• Homogeneity/ heterogeneity

41
Interventions

• Super-realisation bias (Cronbach al. 1980)
• Small-scale interventions tend to get larger
  effects
• Enthusiasm, attention to detail, quality of
  personal relationships

42
Publication bias

• Statistically significant (positive) findings
• Smaller studies need larger effect size to reach
  significance
• Larger effects
• Funnel plot sometimes used to explore this
• Scatterplot of the effects from individual
  studies (horizontal axis) against a study size
  (vertical axis)

43
(No Transcript)
44
Methodological quality

• Traditional reviews privilege methodological
  rigour
• Low quality studies higher effect sizes (Hattie
  Biggs Purdie, 1996)
• No difference (Marzano, 1998)
• High quality studies, higher effect sizes (Lipsey
  Wilson, 1993)
• Depends on your definition of quality

45
(No Transcript)
46
Sample size

• Median effect sizes for studies with sample
  sizes less than 250 were two to three times as
  large as those of larger studies. (Slavin
  Smith, 2008)

47
Heterogeneity

• Variation in effect sizes
• Investigate to find clusters (moderator
  variables)
• Assumption that the effect will be consistent

48
Questions and reactions

• With a colleague see if you can identify a
  question arising from the presentation so far
• What is your reaction to the technique
• How useful is it
• Generally
• To your own work?

49
Strengths of Meta-Analysis

• Uses explicit rules to synthesise research
  findings
• Can find relationships across studies which may
  not emerge in qualitative reviews
• Does not (usually) exclude studies for
  methodological quality to the same degree as
  traditional methods
• Statistical data used to determine whether
  relationships between constructs need clarifying
• Can cope with large numbers of studies which
  would overwhelm traditional methods of review

50
Summary

• Replicable and defensible method for
  synthesizing findings across studies (Lipsey
  Wilson, 2001)
• Identifies gaps in the literature, providing a
  sound basis for further research
• Indicates the need for replication in education
• Facilitates identification of patterns in the
  accumulating results of individual evaluations
• Provides a frame for theoretical critique

51
Other approaches to synthesis

• Narrative
• Quantitative (meta-analysis)
• Best-evidence synthesis (Slavin)
• Realist (Pawson)
• Meta-ethnography (Noblitt Hare)
• Thematic synthesis (Thomas Harden)
• Grounded theory

52
Suggestions

• Be explicit about your rationale
• Be systematic (or at least methodical)
• Be transparent
• Describe
• Analyse (content and methodology)
• Synthesise

53
A (narrative) metaphor

• Literature review as rhetoric
• An act of persuasion
• Introduce your study

54
Some useful websites

• EPPI, Institute of Education, London
• http//eppi.ioe.ac.uk/
• The Campbell Collaboration
• http//www.campbellcollaboration.org/
• Best Evidence Encyclopedia, Johns Hopkins
• http//www.bestevidence.org/
• Best Evidence Synthesis (BES), NZ
• http//www.educationcounts.govt.nz/themes/BES
• Institute for Effective Education (York)
• http//www.york.ac.uk/iee/research/reviews

55
Further training

• ESRC RDI in quantitative synthesis
• One day training sessions
• Introduction (for interpretation)
• Methods Training (for application)
• Issues Seminars (methodological issues)
• Durham, London, Edinburgh, Bristol, Belfast, York
• s.e.higgins_at_durham.ac.uk

56
References

• Bernard, R.M., Abrami, P.C., Lou, Y.,
  Borokhovski, E., Wade, A., Wozney, L., Wallet,
  P.A., Fiset, M., Huang, B. (2004) How Does
  Distance Education Compare with Classroom
  Instruction? A Meta-Analysis of the Empirical
  Literature Review of Educational Research, 74. 3,
  (Autumn, 2004), pp. 379-439.
• Chambers, E.A. (2004). An introduction to
  meta-analysis with articles from the Journal of
  Educational Research (1992-2002). Journal of
  Educational Research, 98, pp 35-44.
• Cronbach, L. J., Ambron, S. R., Dornbusch, S. M.,
  Hess, R.O., Hornik, R. C., Phillips, D. C.,
  Walker, D. F., Weiner, S. S. (1980). Toward
  reform of program evaluation Aims, methods, and
  institutional arrangements. San Francisco, Ca.
  Jossey-Bass.
• Glass, G.V. (2000). Meta-analysis at 25.
  Available at http//glass.ed.asu.edu/gene/papers/
  meta25.html (accessed 9/9/08)
• Hattie, J. A. (1992). Measuring the effects of
  schooling. Journal of Education, 36, pp 5-13
• Hattie, J., Biggs, J. and Purdie, N. (1996)
  Effects of Learning Skills Interventions on
  Student Learning A Meta-analysis Review of
  Educational Research 66.2 pp 99-136.
• Hattie, J.A. (1987) Identifying the salient
  facets of a model of student learning a
  synthesis of meta-analyses International Journal
  of Educational Research, 11 pp 187- 212.
• Hattie, J. Timperley, H. (2007) The Power of
  Feedback Review of Educational Research 77. 1,
  pp. 81112.
• Lipsey, Mark W., and Wilson, David B. (2001).
  Practical Meta-Analysis. Applied Social Research
  Methods Series (Vol. 49). Thousand Oaks, CA SAGE
  Publications.
• Marzano, R. J. (1998) A Theory-Based
  Meta-Analysis of Research on Instruction. Aurora,
  Colorado, Mid-continent Regional Educational
  Laboratory. Available at http//www.mcrel.org80/
  topics/products/83/ (accessed 2/9/08).
• Pearson, D.P., Ferdig, R.E., Blomeyer, R.L.
  Moran, J. (2005) The Effects of Technology on
  Reading Performance in the Middle-School Grades
  A Meta-Analysis With Recommendations for Policy
  Naperville, Il University of Illinois/North
  Central Regional Educational Laboratory .
• Sipe, T. Curlette, W.L. (1997) A Meta-Synthesis
  Of Factors Related To Educational Achievement A
  Methodological Approach To Summarizing And
  Synthesizing Meta-Analyses International Journal
  of Educational Research 25. 7. pp. 583-698.
• Slavin, R.E. and Smith, D. (2008) Effects of
  Sample Size on Effect Size in Systematic Reviews
  in Education Paper presented at the annual
  meetings of the Society for Research on Effective
  Education, Crystal City, Virginia, March 3-4,
  2008.