Evidencebased Software Engineering

1
Evidence-based Software Engineering

• Barbara A. Kittchenham
• Tore Dyba
• Magne Jorgensen
• ICSE 2004
• Poster by Nick Sula

2
Overview

• Purpose of Paper Comparison of organization and
  infrastructure supporting Evidence-Based Medicine
  with the situation in Software Engineering
• Evidence-Based Software Engineering (EBSE)
  encourages integration of research results with a
  view to supporting needs of many stakeholder
  groups
• Results of research The infrastructure needed
  for widespread adoption of EBSE does not
  currently exist
• Conclusion Software Engineering would benefit
  from adopting what it can of the Evidence-based
  approach, provided that it deals with specific
  problems that arise from the nature of SE

3
Evidence-based Medicine

• Change in medical research over the last decade
• In late 80s/early 90s, studies showed failure to
  organize medical research in systematic review
  could cost lives
• Clinical judgement of experts did not compare
  well with results of systematic reviews.
• Evidence-based medicine is the conscientious,
  explicit and judicious use of current best
  evidence in making decisions about the care of
  individual patients
• -Centre for Evidence-Based Medicine
• Adopted in other disciplines Psychiatry,
  nursing, social policy, and education
• Criticism of EBM Applies to populations, not
  necessarily to individuals

4
Goals of Evidence-based SE

• Evidence-based software engineering provides the
  means by which current best evidence from
  research can be integrated with practical
  experience and human values in the decision
  making process regarding the development and
  maintenance of software.
• Establish controls to minimize the risk of
  software systems causing harm
• Adopt appropriate technologies, and avoid
  inappropriate technologies
• EBSE would provide
• Goal for researchers to ensure their work is
  directed to the requirements of industry
• Means by which industry can make rational
  decisions about technology adoption
• Means to improve dependability of software
  intensive systems, via better choice of
  development technologies
• Increase acceptability of software intensive
  systems that interface with individuals
• An input to certification processes

5
EBSE in Practice

• Define an answerable question
• Find the best evidence
• Critically appraise the evidence
• Integrate the appraisal with SE expertise
• Evaluation of the process

6
Implications for EBSE

• Full-scale implementation of EBSE is an extremely
  ambitious goal
• Requires long-term commitment among researchers
  worldwide
• Requires active support from industry
• Requires acceptance within certification bodies

7
Factors specific to SE

• Skill factor
• Medical treatments, though prescribed by skilled
  individuals, do not require skill to administer
  or to receive.
• Experimental protocols
• Blinding Efforts to remove biased expections
  from an experiment
• Lifecycle
• Difficult to isolate techniques
• For example, a design method depends on several
  factors requirements analysis, platform
  language constraints, timescales, and budget.
• Immediate outcomes of a SE technique may not have
  a strong relationship with final project outcomes
• Measures of the design product may have no
  obvious relationship with Application Reliability.

8
Conclusions

• Adoption and use of techniques supported by
  evidence should improve the quality of
  software-intensive systems, and reassure
  stakeholders of best practices
• Requires infrastructure support
• Evaluations of technologies that rely on human
  skill means that experiments are vulnerable to
  subject experimenter bias
• Recommendation Adopt as much of the
  Evidence-based approach as possible, treating
  initial attempts at using EBSE as a means of
  assessing its viability

9
Critique

• Worthwhile examination, especially in the SE
  field which is young and has many rapid
  advancements in technology.
• Adoption of the described process falls victim to
  the same conditions that it attempts to address
• A new area of research in SE needs financial
  support and commitment to establish its necessary
  infrastructure

10
Questions

• 1. Name 3 of the benefits that EBSE would
  provide.
• 2. Name the 5 steps used in applying
  Evidence-Based Software Engineering
• 3. The authors make the following claim
• In most cases software is built with
  technologies for which we have insufficient
  evidence to confirm their suitability, limits,
  qualities, costs, and inherent risks.
• Is this a valid statement, considering the
  current tools and processes that are used to
  track and verify such criteria?

http//csdl.computer.org.ezproxy2.lib.depaul.edu/d
l/proceedings/icse/2004/2163/00/21630273.pdf