Introduction to systematic reviewing 3 May 2005

1

• Evidence-informed education and the work of the
  EPPI-centre
• Josephine Kavanagh
• EPPI-Centre, Social Science Research Unit,
  Institute of Education, London, UK

2
Introduction

• The EPPI-Centre and SSRU
• Why evidence informed practice
• What is a systematic review
• Example EPPI-Review
• Librarians and systematic reviews

3
Social Science Research Unit

• Five streams of work
• Evidence for Policy and Practice Information and
  Co-ordinating Centre (EPPI-Centre)
• Childhood Studies
• Evaluation of Social Interventions
• Sexual Health, Reproduction and Social Exclusion
• Perspectives, Participation and Research
• www.ioe.ac.uk/ssru

4
EPPI-Centre personnel
David Gough (Director), Ann Oakley (Founding
Director) Angela Harden and Sandy Oliver
(Co-directors HP PH) Jo Garcia (User
involvement, Education)
Eva Moran Mark Newman Chloe Powell Rebecca
Rees Abigail Rowe Katy Sutcliffe James
Thomas Carole Torgerson Alex Trouton Helen
Tucker Lisa Underwood
Jon Ashton Mukdarut Bangpan Jackie Barry Ginny
Brunton Jeff Brunton Helen Burchett Esther
Coren Kelly Dickson Adam Fletcher Zoe
Garrett Nicholas Houghton Josephine Kavanagh
5
EPPI-CENTRE VISION AND MISSION STATEMENT

• Developing and promoting participatory and
  user-friendly systematic reviews addressing
  important questions in different domains of
  policy, practice and research in the public
  interest

6
What is a systematic review?

• A systematic review is a piece of research in
  its own right. Beginning with a clearly
  formulated question, it uses systematic and
  explicit methods to identify, select and
  critically appraise relevant research.
• Data from the included studies are extracted,
  analysed and summarised in an appropriate manner
  which may or may not include statistical methods.

7
EPPI-Centre external Review Groups (RGs) and teams

• 16 funded by the Department for Education and
  Skills (DfES) on different aspects of teaching
  and learning (some with joint funding e.g.
  Nuffield Foundation, NUT (HEFCE), ESRC TLRP)
• 8 funded by the Teacher Training Agency on
  different aspects of teacher education
• 1 on transitions of students with disabilities
  from school to work funded by (based in Colorado
  in the US)
• 1 in higher education funded by the Learning
  Skills Development Agency
• New collaborations with SCIE and DWP

8
EPPI-Centre in house reviews

• The effect of travel modes on children's mental
  health, cognitive, social development (DETR)
• The effects of personal development profiling for
  improving student learning (LTSN)
• Support for pupils with emotional and behavioural
  difficulties (NFER)
• Effect of school size on student attainment
  (DfES)
• Series of reviews on health promotion and public
  health (HP PH) topics funded primarily by the
  Department of Health England

9
Health promotion public health review series
19971999 Workplace health promotion Peer-delivere
d health promotion for young people

• 19931996
• Young people and smoking
• Older people and accidents
• Young people and sexual health
• Men who have sex with men (MSM) and sexual health

19992001 Young people and mental
health physical activity healthy eating
20012004 Children and physical
activity healthy eating HIV-health promotion for
MSM
20042007 Incentives for young people Active
transport Teenage pregnancy and parenting Risk
behaviour and accidental injury
10
EPPI-Centre Databases

• Web site for all EPPI-Centre reviews
  http//eppi.ioe.ac.uk/
• Research Evidence in Education Library (REEL)
• Web-page accessibility
• Review Group details
• Protocols
• Reviews and user perspectives
• Primary data underlying reviews
• Bibliographic information
• Detailed descriptions and quality assessment

11
Research for Policy and Practice

• It should be self-evident that decisions on
  Government policy ought to be informed by sound
  evidence. Social science ought to be
  contributing a major part of that evidence
  baseToo often in the past policy has not been
  informed by good research a former Permanent
  Secretary once ruefully described the old DES as
  a knowledge-free-zone.
• David Blunkett, February 2000

12
Research evidence for policy and practice

• policy makers and practitioners who intervene
  in the lives of other people not infrequently do
  more harm than good
• Chalmers I (2003) Trying to do more good than
  harm in policy and practice the role of
  rigorous, transparent, up to date, replicable
  evaluation. Paper commissioned for the Annals of
  the American Academy of Political and Social
  Science.

13
More harm than good?
14
More harm than good?
15
Reduce the Risk Campaign in the early 1990s in
the UK

• The risk of cot death is reduced if babies are
  NOT put on the tummy to sleep. Place your baby on
  the back to sleep. ..Healthy babies placed on
  their backs are not more likely to choke.

16
Reduce the Risk Campaign in the early 1990s in
the UK
17
Need for reviews

• UK government proposed to introduce driver
  education programmes in schools and colleges.
  Recent systematic review suggests it may lead to
• Early licensing
• a modest but potentially important increase in
  the proportion of teenagers involved in traffic
  crashes
• Roberts, Kwan and the Cochrane Injuries Group
  Driver Education Reviewers. School based driver
  education for the prevention of traffic crashes
  (Cochrane Review). In The Cochrane Library,
  Issue 4, 2002. Oxford Update Software.

18
Educational research for practice
. . . (which is) presented in a form or medium
which is largely inaccessible to a non-academic
audience and lack(s) interpretation for a policy
making or practitioner audience. Hillage J,
Pearson R, Anderson A, Tamkin P (1998) Excellence
in Research in Schools. London Department for
Education and Employment/ Institute of
Employment Studies
19
Facilitating the use of research for policy and
practice

• By preparing systematic reviews of the results
  of relevant, reliable researchAdditional
  research, IF systematic reviews of existing
  research show that this is needed

20
Need for systematic reviews

• Before any new policy, practice, research
  sensible to ask what is already known within
  particular conceptual frameworks and contexts.
• To clarify best practice in areas where there is
  uncertainty
• Contextualizes information from new studies
• Involves explicit systematic methods and thus
  transparency
• These may be lacking in non systematic reviews
  and expert opinion however excellent

21
Key features of a systematic review

• Synthesises the results of primary research
• Uses explicit and transparent method
• A piece of research, following standard set of
  stages
• Accountable, replicable, updateable
• Need for user involvement

22
Not just any reviews

• 6 reviews of older people and accident
  prevention
• Total studies included 137
• Common to at least two reviews 33
• Common to all six reviews 2
• Treated consistently in all reviews 1
• Oliver S (1999) Users of health services
  following their agenda. In Hood S, Mayall B,
  Oliver S (eds.) Critical Issues in Social
  Research Power and Prejudice. Buckingham Open
  University Press.

23
Stages of an EPPI-Centre review
MAP
Applying inclusion and exclusion criteria to
studies
IN-DEPTH REVIEW
24
School size review
25
Diagrammatic summary of relationship between
school size and outcomes
26
Possible evidence based statement

• Bigger schools have better exam results and cost
  less
• but are students and staff less happy?

27
Further details in

• Garrett Z, Newman M, Elbourne D, Bradley S, Noden
  P, Taylor J, West A (2004) Secondary School Size
  A Systematic Review. In Research Evidence in
  Education Library. London EPPI-Centre, Social
  Science Research Unit, Institute of Education,
  University of London.
• http//eppi.ioe.ac.uk/EPPIWebContent/reel/review_g
  roups/School_size/s_s_rv1.pdf
• Newman M, Garrett Z, Elbourne D, Bradley S Noden
  P, Taylor J, West A. Does secondary school size
  make a difference? A systematic review
  (submitted)

28
Roles for the librarian in a systematic review
Applying inclusion and exclusion criteria to
studies
29
Systematic review sources

• Research Evidence in Education Library (REEL)
  http//eppi.ioe.ac.uk/EPPIWeb/home.aspx?page/reel
  /intro.htm
• Current Educational Research in the UK (CERUK)
  http//195.194.2.38/starweb/ceruk/servlet.starweb?
  pathceruk/CERUK.web
• CAMPBELL COLLABORATION - C2 RIPE
  http//www.campbellcollaboration.org/Fralibrary2.h
  tml
• ERIC search for systematic reviews or
  meta-analyses
• Other social science / educational databases

30
Developments

• Online journals and public access - resource
  implications
• Research visualisation software e.g RefViz
• Availability and development of online
  systematic review software such as
  EPPI-Reviewer
• Search methods developing, particularly in
  locating qualitative research
• E.g. Comprehensive searching vs purposive sampling

31
THANK YOU j.kavanagh_at_ioe.ac.uk