Software Engineering Research Approaches

1
Software Engineering Research Approaches

• Balancing theory and praxis
• How engineering research differs from scientific
  research
• The role of empirical studies
• Models for SE research

2
The need to link research with practice
Colin Potts, Software Engineering Research
Revisited, IEEE Software, September 1993

• Why after 25 years of SE has SE research failed
  to influence industrial practice and the quality
  of resulting software?
• Potts argues that this failure is caused by
  treating research and its application by industry
  as separate, sequential activities.
• What he calls the research-then-transfer
  approach. The solution he proposes is the
  industry-as-laboratory approach.
• .

3
Research-then-Transfer




Problem V1
Research Solution V1
Wide gulf bridged by indirect, anecdotal knowledge
Problem V2
Research Solution V2
Problem V3
Research Solution V3
Technology transfer Gap bridged by hard, but
frequently inappropriate technology
Problem V4
Research Solution V4
Incremental Refinement of research solutions
Problem evolves
invisibly to the
research community
4
Research-then-Transfer Problems

• Both research and practice evolve separately
• Match between current problems in industry and
  research solutions is haphazard
• No winners

5
Disadvantages of Research-then-Transfer

• Research problems described and understood in
  terms of solution technology - whatever is
  current research fashion. Connection to practice
  is tenuous.
• Concentration is on technical refinement of
  research solution - OK but lacks industrial need
  as focus, so effort may be misplaced.
• Evaluation is difficult as research solutions may
  use technology that is not commonly used in
  industry
• Delay in evaluation means problem researchers are
  solving has often evolved through changes in
  business practice, technology etc.
• Transfer is difficult because industry has little
  basis for confidence in proposed research
  solution.

6
Industry-as-Laboratory Approach to SE research
Problem V1
Research Solution V1
Problem V2
Research Solution V2
Problem V3
Research Solution V3
Problem V4
Research Solution V4
7
Advantages of Industry-as-Laboratory Approach

• Stronger connection at start because knowledge of
  problem is acquired from the real practitioners
  in industry, often industrial partners in a
  research consortium.
• Connection is strengthened by practitioners and
  researchers constantly interacting to develop the
  solution
• Early evaluation and usage by industry lessens
  the Technology Transfer Gap.
• Reliance on Empirical Research
• shift from solution-driven SE to problem-focused
  SE
• solve problems that really do matter to
  practitioners

8
Early SEI industrial survey research

• What a SEI survey learned from industry
• There was a thin spread of domain knowledge in
  most projects
• Customer requirements were extremely volatile.
• These findings point towards research combining
  work on requirements engineering with reuse -
  instead of the approach of researching these
  topics by separate SE research communities - as
  is still found today!
• From A field study of the Software Development
  Process
• for Large Systems, CACM, November 1988.

9
Further Results from Potts et al Early 90s Survey

• 23 software development organizations (during
  1990-92). (Survey focused on Requirements
  Modeling process)
• Requirements were invented not elicited.
• Most development is maintenance.
• Most specification is incremental.
• Domain knowledge is important.
• There is a gulf between the developer and user
• User-interface requirements continually change.
• There is a preference for office-automation tools
  over CASE tools to support development. I.e.
  developers found using a WP DB more useful
  than any CASE tools.

10
Industry-as-Laboratory emphasizes Real Case
Studies

• Advantages of case studies over studying problems
  in research lab.
• Scale and complexity - small, simple (even
  simplistic) cases avoided - these often bear
  little relation to real problems.
• Unpredictability - assumptions thrown out as
  researchers learn more about real problems
• Dynamism - a real case study is more vital than
  a textbook account
• The real-world complications of industrial case
  studies are more likely to throw up
  representative problems and phenomena than
  research laboratory examples influenced by the
  researchers preconceptions.

11
Need to consider Human/Social Context in SE
research

• Not all solutions in software engineering are
  solely technical.
• There is a need to examine organizational, social
  and cognitive factors systematically as well.
• Many problems are people problems, and require
  people-orientated solutions.

12
Theoretical SE research

• While there is still a place for innovative,
  purely speculative research in Software
  Engineering, research which studies real problems
  in partnership with industry needs to be given a
  higher profile.
• These various forms of research ideally
  complement one another.
• Neither is particularly successful if it ignores
  the other.
• Too industrially focused research may lack
  adequate theory!
• Academically focused research may miss the
  practice!

13
Research models for SE

• Problem highlighted by Glass
• Most SE Research in 1990s was Advocacy
  Research. Better research models needed.
• The software crisis provided the platform on
  which most 90s research was founded.
• SE Research ignored practice, for the most part
  lack of practical application and evaluation were
  gapping holes in most SE research.
• Appropriate research models for SE are needed.
• Robert Glass, The Software -Research Crisis,
  IEEE Software, November 1994

14
Methods underlying Models

• Scientific method
• Engineering method
• Empirical method
• Analytical method
• From W.R.Adrion, Research Methodology in Software
  Engineering, ACM SE Notes, Jan. 1993

15
Scientific method

Observe real world
Propose a model or theory of some real world
phenomena
Measure and analyze above
Validate hypotheses of the model or theory
If possible, repeat
16
Engineering method

Observe existing solutions
Propose better solutions
Build or develop better solution
Measure, analyze, and evaluate
Repeat until no further improvements are possible
17
Empirical method


Propose a model
Develop statistical or other basis for the model
Apply to case studies
Measure and analyze
Validate and then repeat
18
Analytical method



Propose a formal theory or set of axioms
Develop a theory
Derive results
If possible, compare with empirical observations
Refine theory if necessary
19
Need to move away from purely analytical method

• The analytical method was the most widely used in
  mid-90s SE research, but the others need to be
  considered and may be more appropriate in some SE
  research.
• Good research practice combines elements on all
  these approaches.

20
4 important phases for any SE research project
(Glass)

• Informational phase - Gather or aggregate
  information via
• reflection
• literature survey
• people/organization survey
• case studies
• Propositional phase - Propose and build
  hypothesis, method or algorithm, model, theory or
  solution
• Analytical phase - Analyze and explore proposal
  leading to demonstration and/or formulation of
  principle or theory
• Evaluation phase - Evaluate proposal or analytic
  findings by means of experimentation (controlled)
  or observation (uncontrolled, such as case study
  or protocol analysis) leading to a substantiated
  model, principle, or theory.

21
Software Engineering Research Approaches

• The Industry-as-Laboratory approach links theory
  and praxis
• Engineering research aims to improve existing
  processes and/or products
• Empirical studies are needed to validate Software
  Engineering research
• Models for SE research need to shift from the
  analytic to empirical.