EvidenceBased Software Engineering and Systematic Reviews

1
Evidence-Based Software Engineeringand
Systematic Reviews

• Barbara Kitchenham

2
Agenda

• The evidence-based paradigm
• Evidence-Based Software Engineering (EBSE)
• Systematic Reviews

3
The Evidence-Based Paradigm

• Evidence-based medicine has changed research
  practices
• Medical researchers found
• Failure to organise existing medical research
  cost lives
• Clinical judgement of experts worse than
  systematic reviews
• Evidence-based paradigm adopted by many other
  disciplines providing service to public
• Social policy
• Education
• Psychiatry

4
Goal of EBSE

• EBM Integration of best research evidence with
  clinical expertise and patient values
• EBSE Adapted from Evidence-Based Medicine
• To provide the means by which current best
  evidence from research can be integrated with
  practical experience and human values in the
  decision making process regarding the development
  and maintenance of software
• Anticipated benefits
• Common goals for research groups
• Help for practitioners adopting new technologies
• Means to improve dependability
• Increase acceptability of software-intensive
  systems
• Input to certification process

5
Practicing EBSE

• Convert information need into answerable question
• Track down best evidence
• Critically appraise evidence
• Integrate critical appraisal with SE expertise
  and stakeholder requirements
• Evaluate and improve above steps

6
Systematic Reviews - 1/2

• A systematic review is
• An overview of research studies that uses
  explicit and reproducible methods
• Systematic reviews aim to synthesise existing
  research
• Fairly (without bias)
• Rigorously (according to a defined procedure)
• Openly (ensuring that the review procedure is
  visible to other researchers)

7
Systematic Reviews 2/2

• Support Evidence-based paradigm
• Start from a well-defined question
• Step 1
• Define a repeatable strategy for searching the
  literature
• Step 2
• Critically assess relevant literature
• Step 3
• Synthesise literature
• Step 4 (but only partially)

8
Advantages

• Provide information about effects of a phenomenon
  across wide range of settings
• Essential for SE where we have sampling problems
• Consistent results provide evidence that
  phenomena are
• Robust
• Transferable
• Inconsistent results
• Allow sources of variation to be studied
• Meta-analysis possible for quantitative studies

9
Anticipated Benefits

• Create a firm foundation for future research
• Position your own research in the context of
  existing research
• Close areas where no further research is
  necessary
• Uncover areas where research is necessary
• Help the development of new theories
• Identify common underlying trends
• Identify explanations for conflicting results
• Should be a standard research methodology

10
Disadvantages

• Require more effort than informal reviews
• Difficult for lone researchers
• Standards require two researchers
• Minimising individual bias
• Incompatible with requirements for short papers

11
Value of Systematic Reviews

• Can contradict common knowledge
• Jørgensen and Moløkken reviewed surveys of
  project overruns
• Standish CHAOS report is out of step with other
  research
• May have used inappropriate methodology
• Jørgensen reviewed evidence about expert opinion
  estimates
• No consistent support for view that models are
  better than human estimators

12
Systematic Review Process
Develop Review Protocol
Plan Review
Validate Review Protocol
Identify Relevant Research
Select Primary Studies
Conduct Review
Assess Study Quality
Extract Required Data
Synthesise Data
Document Review
13
Developing the Protocol

• Review protocol
• Specifies methods to be used for a systematic
  review
• Predefined protocol
• Reduces researcher bias by reducing opportunity
  for
• Selection of papers driven by researcher
  expectations
• Changing the research question to fit the results
  of the searches
• Good practice for any empirical study

14
Protocol Contents -1/2

• Background
• Rationale for survey
• Research question
• Critical to define this before starting the
  research
• Strategy used to search for primary sources
• Individual studies of the phenomenon of interest

15
Protocol Contents 2/2

• Strategy to find primary studies
• Search terms, resources, databases, journals,
  conferences
• Procedures for storing references
• How publication bias will be handled
• Grey literature
• Direct approach to active researchers
• How completeness will be determined
• Useful to have the baseline paper to set start
  date
• Selection Strategy
• Inclusion/exclusion criteria
• Handling multiple papers on one experiment

16
Protocol Contents- 2/3

• Quality assessment criteria
• Criteria used to evaluate quality of primary
  sources
• Data extraction
• What data will be extracted from each primary
  source
• How to handle missing information
• How data reliability will be addressed
• Usually multiple reviewers
• Where data will be stored
• Procedures for data synthesis
• Formats for summarising data
• Measures and analysis if meta-analysis is
  proposed
• Should tested during protocol construction

17
Research Question 1/2

• Question types for EBSE
• Assessing the effect of an SE technology
• Assessing the frequency or rate of a project
  development factor
• E.g. Rate of project failures
• Identifying cost and risk factors
• Identifying impact of technology on reliability,
  performance, cost
• Possible to have more general questions for other
  purposes
• Review of research in software engineering
  (Glass, et al., 2002)

18
Research Question 2/2

• Question structure
• Population
• People, projects types, applications types
  affected by the intervention
• Intervention
• Software method, tool, procedure
• Outcomes
• Impact of technology in terms relevant to
  practitioners
• Cost, quality, time to market
• Experimental designs
• Any constraints on type of primary studies to be
  included

19
Next steps are easy!?

• Conduct the review
• Enact the protocol
• Expect further iterations of
• Search strategy
• Selection criteria
• Data extraction
• Record any deviations from protocol
• Document the Review
• Using procedures defined in protocol

20
Conclusions

• Evidence-based approach
• Revolutionised medicine
• May be relevant to SE
• Systematic reviews
• Support the evidence-based approach
• Valuable as a research tool
• Even if we dont accept EBSE

21
References
Australian National Health and Medical Research
Council. How to review the evidence systematic
identification and review of the scientific
literature, 2000. IBSN 186-4960329 . Australian
National Health and Medical Research Council. How
to use the evidence assessment and application
of scientific evidence. February 2000, ISBN 0 642
43295 2. Cochrane Collaboration. Cochrane
Reviewers Handbook. Version 4.2.1. December
2003. Glass, R.L., Vessey, I., Ramesh, V.
Research in software engineering an analysis of
the literature. IST 44, 2002, pp491-506 Magne
Jørgensen and Kjetil Moløkken. How large are
Software Cost Overruns? Critical Comments on the
Standish Groups CHAOS Reports,
http//www.simula.no/publication_one.php?publicati
on_id711, 2004. Magne Jørgensen. A Review of
Studies on Expert Estimation of Software
Development Effort. Journal Systems and Software,
Vol 70, Issues 1-2, 2004, pp 37-60.
22
References
Khan, Khalid, S., ter Riet, Gerben., Glanville,
Julia., Sowden, Amanda, J. and Kleijnen, Jo.
(eds) Undertaking Systematic Review of Research
on Effectiveness. CRDs Guidance for those
Carrying Out or Commissioning Reviews. CRD Report
Number 4 (2nd Edition), NHS Centre for Reviews
and Dissemination, University of York, IBSN 1
900640 20 1, March 2001. Kitchenham, Barbara.
Procedures for Performing Systematic Reviews,
Joint Technical Rreport, Keele University
TR/SE-0401 and NICTA 0400011T.1, July 2004. Pai,
Madhukar, McCullovch, Michael, Gorman, Jennifer
D., Pai, Nitika, Enanoria, Wayne, Kennedy, Gail,
Tharyan, Prathap, Colford, John M. Jnr.
Systematic reviews and meta-analysis An
illustrated, step-by-step guide. The National
medical Journal of India, 17(2) 2004, pp
86-95. Sackett, D.L., Straus, S.E., Richardson,
W.S., Rosenberg, W., and Haynes, R.B.
Evidence-Based Medicine How to Practice and
Teach EBM, Second Edition, Churchill Livingstone
Edinburgh, 2000.