Evidence-Based Software Engineering: A Paradigm for the Future?

1
Evidence-Based Software Engineering A Paradigm
for the Future?

• Tore Dybå
• Chief Scientist
• SINTEF ICT
• tore.dyba_at_sintef.no
• JavaZone, Oslo, 15 September 2005

2
Outline

• The problem
• How do I know that what I know is true?
• An example from object-oriented design
• The evidence-based paradigm
• Evidence-Based Software Engineering (EBSE)
• Practical and scientific challenges
• Conclusion
• Bibliography

3
The problem

• Software development organizations are employing
  methods and tools that frequently lack sufficient
  evidence regarding their suitability, limits,
  qualities, costs, and inherent risks.
• Heres a message from software practitioners to
  software researchers
• We need your help. We need some better advice on
  how and when to use methodologies. (Robert L.
  Glass, Communications of the ACM, vol. 47, no. 5,
  May 2004, pp. 19-21).

4
How do I know that what I know is true?

• There are three basic answers to this question
• Authority truth is given to us by someone more
  knowledgeable than ourselves
• Omniscient authority (the authority is God)
• Human authority (the authority is a human expert)
• Reason what is true is that which can be proven
  using the rules of deductive logic
• Experience what is true is that which can be
  encountered through one or more of the senses
• Anecdotal experience
• Empirical evidence

5
Relative strengths of these approaches

• Omniscient authority provides absolute truth
• if there is a God and He has spoken on something,
  
• then what He says must, by definition, be true.
• Reason yields conditional absolute truth
• if the premises of a valid deductive argument are
  known to be true,
• then the conclusion of the argument must also be
  true.
• Empirical evidence provides probable truth
• if controlled experiments are designed properly
  and repeated enough times,
• then it is highly probable that the results
  accurately describe reality.
• Anecdotal experience yields possible truth
• if something happened for one person,
• it is possible it might happen to others also.
• Finally, human authority provides opinion.

6
On which approaches is SE mostly based?

• The software engineering literature is filled
  with pronouncements about how software should be
  developed.
• Representative comments include the following
• The best way to develop reusable software is to
  use object-oriented design
• To create reliable software, you must use
  n-version programming
• The only way to ensure reliable software is to
  use formal methods
• Rapid prototyping is the best way to get the
  specifications right
• Functional programming is the only way to go
• GOTO's are harmful
• A developer is unsuited to test his or her code
• Adding manpower to a late project makes it later

7
Delegated vs. Centralized control expert
opinions

• The Delegated Control Style
• Rebecca Wirfs-Brock A delegated control style
  ideally has clusters of well defined
  responsibilities distributed among a number of
  objects. To me, a delegated control architecture
  feels like object design at its best
• Alistair Cockburn The delegated coffee-machine
  design is, I am happy to see, robust with
  respect to change, and it is a much more
  reasonable ''model of the world.'
• The Centralized Control Style
• Rebecca Wirfs-Brock A centralized control style
  is characterized by single points of control
  interacting with many simple objects. To me,
  centralized control feels like a "procedural
  solution" cloaked in objects
• Alistair Cockburn Any oversight in the
  mainframe object (even a typo!) in the
  centralized coffee-machine design means
  potential damage to many modules, with endless
  testing and unpredictable bugs.

8
Evaluating the Effect of a Delegated versus
Centralized Control Style on the Maintainability
of Object-Oriented Software
Assuming that it is not only highly skilled
experts who are going to maintain an
object-oriented system, a viable conclusion from
the controlled experiment reported in this paper
is that a design with a centralized control style
may be more maintainable than is a design with a
delegated control style.
Erik Arisholm and Dag Sjøberg, IEEE Transactions
on Software Engineering, vol. 30, no. 8, August
2004, pp. 521-534.
9
Pair programming leads to faster, better, and
cheaper software or
10
The Pair Programming Experiment

• 295 junior, intermediate and senior professional
  Java consultants in total
• 99 individuals (conducted in 2001)
• 98 pairs (conducted in 2004/2005)
• Norway 41
• Sweden 28
• UK 29

Hans Gallis, Erik Arisholm, Tore Dybå and Dag
Sjøberg, Work in progress, Simula Research
Laboratory and SPIKE project (NFR).
11
Junior developers benefit most from pair
programming, in particular on complex tasks!

• The effect of pair programming on duration,
  effort and correctness depends on the expertise
  of the developers and the type of task to be
  solved
• The benefits of pair programming are reduced with
  increasing programming skills of the individuals
• The benefits of pair programming are reduced with
  decreasing task complexity
• Pair programming requires significant more effort
  than individual programming (regardless of
  programmer category)
• Juniors
• Pair programming significantly improves
  correctness (less logical errors)
• The effect of pair programming on correctness is
  highest for the tasks based on the delegated
  control style
• Junior pairs obtain almost the same correctness
  as intermediate and senior pairs ( 80 percent
  correct)
• Seniors
• No clear benefits of pair programming

Hans Gallis, Erik Arisholm, Tore Dybå and Dag
Sjøberg, Work in progress, Simula Research
Laboratory and SPIKE project (NFR).
12
Archie Cochrane
It is surely a great criticism of our
profession that we have not organised a critical
summary, by specialty or subspecialty, adapted
periodically, of all relevant randomised
controlled trials.
13
The Evidence-Based Paradigm

• Evidence-Based Medicine (EBM) has changed
  research practices
• Medical researchers found
• Failure to organize existing medical research
  cost lives
• Clinical judgment of experts worse than
  systematic reviews
• Evidence-based paradigm adopted by many other
  disciplines providing service to public
• Social policy
• Education
• Psychiatry

D.L. Sackett, S.E. Straus, W.S. Richardson, W.
Rosenberg, and R.B. Haynes, Evidence-Based
Medicine How to Practice and Teach EBM, Second
Edition, Churchill Livingstone Edinburgh, 2000.
14
What is evidence?

• Systematic reviews
• Methodologically rigorous synthesis of all
  available research relevant to a specific
  research question
• Not ad hoc literature reviews
• Best systematic reviews based on Randomized
  Controlled Trials (RCTs)
• Not laboratory experiments
• Trials of real treatments on real patients in a
  clinical setting

15
Integrating evidence

• Medical researchers practitioners construct
  practitioner-oriented guidelines
• Assess the evidence
• Determine strength of evidence (type of study)
• Size of effects (practical not just statistical)
• Relevance (appropriateness of outcome measures)
• Assess applicability to other settings
• Summarize benefits harms
• Present the evidence to stakeholders
• Balance sheet of evidence harms

16
Goal of EBSE

• EBM Integration of best research evidence with
  clinical expertise and patient values
• EBSE Adapted from Evidence-Based Medicine
• To provide the means by which current best
  evidence from research can be integrated with
  practical experience and human values in the
  decision making process regarding the development
  and maintenance of software
• EBSE sets requirements on practitioners and
  researchers
• Practitioners need to track down and use best
  evidence in context of practice
• Researchers need to provide best evidence

17
The steps of EBSE

• EBSE is a process involving five steps
• Converting a relevant problem or information need
  into an answerable question.
• Searching the literature for the best available
  evidence to answer the question.
• Critically appraising the evidence for its
  validity, impact, and applicability.
• Integrating the appraised evidence with practical
  experience and the values and circumstances of
  the customer to make decisions about practice.
• Evaluating performance and seeking ways to
  improve it.

18
Step 1 Asking an answerable question

• The first step in EBSE is to convert a relevant
  problem or information need into an answerable
  question.
• Typical questions ask for specific knowledge
  about how to appraise and apply methods, tools,
  and techniques in practice.
• Well formulated questions usually have three
  components
• The main intervention or action we are interested
  in.
• The context or specific situations of interest.
• The main outcomes or effects of interest.
• Example
• Does the use of pair programming lead to
  improved code quality when practiced by
  professional software developers?

19
Step 2 Finding the best evidence

• Finding an answer to our question includes
  selecting an appropriate information resource and
  executing a search strategy.
• The main source of research-based evidence is
  articles published in scientific journals.
  Examples of databases that index published
  articles include
• IEEE Xplore, http//ieeexplore.ieee.org
• ACM Digital Library, http//www.acm.org/dl
• ISI Web of Science, http//isiknowledge.com
• Often, reading important magazines such as the
  Communications of the ACM, IEEE Computer, IEEE
  Software, and IT Professional would probably be
  enough to get a general overview of the latest
  develop-ments within software engineering.

20
Step 3 Critically appraising the evidence

• Unfortunately, published research isnt always of
  good quality the problem under study might be
  unrelated to practice or the research method
  could have weaknesses so that the results cannot
  be trusted.
• To assess whether research is of good quality and
  can be applied to practice, we must be able to
  critically appraise the evidence.
• Is there any vested interest?
• Is the evidence valid?
• Is the evidence important?
• Can the evidence be used in practice?
• Is the evidence in this study consistent with the
  evidence in other available studies?

21
Step 4 Applying the evidence

• Active use of new knowledge is characterized by
  applying or adapting specific evidence to a
  specific situation in practice.
• Therefore, in order to practice EBSE, the
  individual software developer must commit him or
  herself to actively engage in a learning process,
  combining the externally transmitted evidence
  with prior knowledge and experience.
• Thus, it is at this point that EBSE needs to be
  integrated with process improvement.
• EBSE should provide the scientific basis for
  undertaking specific process changes while SPI
  should manage the process of introducing a new
  technology.

22
Step 5 Evaluating performance

• We need to consider how well we perform each step
  in the EBSE process and how we might improve our
  use of EBSE.
• In particular, we should ask ourselves how well
  we are integrating evidence with practical
  experience, customer requirements, and our
  knowledge of the specific circumstances.
• Following SPI practice, we also need to assess
  whether process change has been effective.
• This might include After Action Reviews,
  Postmortem Analyses, and organization-wide
  measurement programs.

23
Software engineering challenges

• No comparable (to medicine) research
  infrastructure.
• No agreed standards for empirical studies
• A proposal for formal experiments and surveys
• Nothing for qualitative or observational studies
• No agreed standards for systematic review
• Few software engineering guidelines based on
  empirical evidence.
• Challenges in addressing software engineering
  specifics
• The skill factor
• The lifecycle issue
• The context dependences

24
Is evidence worth waiting for?
Scott Ambler, Answering the "Where is the Proof
That Agile Methods Work" Question,
http//www.agilemodeling.com/essays/proof.htm
25
Conclusion

• Evidence-based practice works in medicine.
• Experience from undertaking empirical studies,
  systematic reviews, and teaching students in EBSE
  gives some confidence that it will work within
  software engineering as well.
• However, EBSE lacks the infrastructure required
  to support the evidence-based paradigm
• Would need financial support to put in place
  appropriate infrastructure
• Need to develop appropriate protocols for SE
  studies
• Some aspects of EBSE are easy to adopt, e.g.
  systematic reviews
• EBSE needs to be tested on real and relevant
  problems
• Guidelines for practice based on systematic
  reviews
• Improvement strategies need to take more than
  scientific evidence into consideration and must
  be balanced according to your specific situation.

26
Bibliography

• Barbara A. Kitchenham, Tore Dybå and Magne
  Jørgensen, Evidence-Based Software Engineering,"
  Proceedings of the 26th International Conference
  on Software Engineering (ICSE 2004), Edinburgh,
  Scotland, 23-28 May, IEEE Computer Society, 2004,
  pp. 273-281.
• Tore Dybå, Barbara A. Kitchenham and Magne
  Jørgensen, Evidence-Based Software Engineering
  for Practitioners, IEEE Software, vol. 22, no.
  1, January/February, 2005, pp. 58-65.
• Magne Jørgensen, Tore Dybå and Barbara A.
  Kitchenham, Teaching Evidence-Based Software
  Engineering to University Students," Proceedings
  of the 11th International Software Metrics
  Symposium (METRICS 2005), Como, Italy, 19-22
  September, 2005.
• D.L. Sackett, S.E. Straus, W.S. Richardson, W.
  Rosenberg, and R.B. Haynes, Evidence-Based
  Medicine How to Practice and Teach EBM, Second
  Edition, Churchill Livingstone Edinburgh, 2000.