Science

1
Science Substance A Challenge to Software
Engineers

• Chantel Bowie

2
The Gist

• Software Engineer (SE) Researchers
• Improve software development and maintenance with
  new practices
• Backed by hard evidence
• Substance vs. Science
• Science is better

3
Research Claims

• SE Researchers
• Examine carefully through scientific
  experimentation
• Scientific way to design and carry out software
  engineering investigations

4
Research Realities

• SE Research
• 5 Questions should be asked
• 1. Is it based on empirical evaluation and data?
• 2. Was the experiment designed correctly?
• 3. Is it based on a toy or a real situation?
• 4. Were the measurements used appropriate to the
  goals of the experiment?
• 5. Was the experiment run for a long enough time?

5
Evaluation Data

• Is it based on empirical evaluation and data?
• Realistic project with realistic objects?
• Benefits identified MUST be clearly derived from
  concept in question
• Hypothesis
• Disadvantage
• Time consuming
• Little is actually being done

6
Design

• Was the experiment designed correctly
• Design must be correct for the hypothesis in
  question
• Case Studies
• Ben Shneiderman
• Flow charts DO NOT HELP programmers comprehend
  documentation any better than pseudocode
• David Scanlan
• Demonstrated structured flowcharts are preferable
  to pseudocode for program documentation

7
Design

• Was the experiment designed correctly
• Software Engineers not familiar with how to
  establish/evaluate proper design
• Guidelines in production
• Evaluate effectiveness of standards and methods
• Experimental design and statistical data

8
Toy vs. Real

• Is it based on a toy or a real situation?
• Large scale projects (Real)
• High design costs
• SE research forced to conduct artificial problems
  in artificial situations
• Smaller scale projects (Toy)
• Smaller research projects better
• Team cooperation, financial backing

9
Measuring

• Were the measurements used appropriate to the
  goals of the experiment?
• May be designed properly
• May measure and analyze insufficient data or
  wrong data
• Attributes to measure
• Maintainability
• Reliability

10
Long Term

• Was the experiment run for a long enough time?
• Sometimes its just not long enough
• Short term results masquerade as long term
  effects
• Take long term view of practices promising
  profound effects on development and maintenance

11
Conclusion

• New technologies, tools, methods
• Little evidence radically improves the way we
  develop software systems

• Challenge to SE community
• Software manager
• Insist on qualitative data, well-designed
  experimental research
• Be willing to participate in such experiments
• Software developer/maintainer
• Participate in new experiments involving existing
  or new methods