Progress in Testing Component-Based Software

1
Progress in Testing Component-Based Software

• Craig H. Wittenberg
• Component Applications Group
• Microsoft Research

2
Talk Outline

• Our project
• Demos
• Component and composition
• Progress in testing components
• Areas for further research

3
High Level Goals

• A system of flexible components for a family of
  products
• Faster and cheaper application assembly and
  evolution
• Reliable, efficient, and consistent applications
  working together

4
Octarine Demo

• Prototype written in C
• Looks familiar
• Adding Bullets, Music
• Followed by architectural drawings

5
Octarine Text Architecture
Text
ISequence
Story
ISequence
Music
ISequence
Table
ISequence
Text
6
Adding Bullets/Numbering
Text
ISequence
Story
ISequence
Music
ISequence
ISequence
Text
Bullets/ Numbering
ISequence
Table
ISequence
Text
7
Landscape Demo

• More recent
• Many small components
• Static and dynamic connections
• Multiple uses of
• Text, Layout, Grid, Data pipes
• Architecture for many Uis
• Small devices
• Non-visual interfaces

8
Our Approach

• Factoring and composition
• Precise interface specs
• Significant design and prototype step
• Huge emphasis on testing
• Control inter-component dependencies
• Tools support (design, build, and run time)
• Critical mass of designs, components, frameworks,
  large demos

9
Component
A software component is a unit of composition
with contractually specified interfaces and
explicit context dependencies only. A software
component can be deployed independently and is
subject to composition by third
parties. Szyperski, Component Software, pg. 34.

• Key points Contracts, Composition, Dependencies,
  Deployment
• No observable global state

10
Component DLL

• Binary package plus docs
• No top-level APIs
• Contains one or more classes
• Implementation of contracts only
• No implementation inheritance
• Realize behavior of component
• Instances of classes are connected
• Dependencies upon
• other classes
• services from other instances

11
Precise Interface Specs a.k.a. Software Contracts

• An interface has
• Name (compile time and runtime)
• Methods with pre/post conditions
• Abstract model with invariants
• State table
• Specified separately from implementation
• Implementation does the obvious and
• maps model variables to internal state
• Never changed once published

12
ILight Methods and Model

• ILight, 2461A1A0-639E-11d2-874D-00AA0060FCA9
• TurnOn()
• TurnOff()
• SetIntensity(int desired)
• Model
• boolean on
• int intensity // range is 0..100.

13
ILight Pre/Post Conditions

• TurnOn()
• Post on true.
• TurnOff()
• Post on false.
• SetIntensity(desired)
• Pre desired gt0 desired lt 100.
• Post intensity desired.

14
A Light

• The Light class implements ILight
• Contract-specific and implementation-specific
  model mapping

Light
ILight
Map ILight model to Impl.
15
Record Filter

• Filter processes one set of records into another

Plants
IRecordSet
IRecordSet
Record Filter
IString
Filter Spec
Map IRecordSet model to Impl.
16
Light Switch

• Converts calls to push() into on() and off()

IPushIt
ILight
Light Switch
A light
Map IPushIt model to Impl.
17
Closure

• Executes closure over parameters
• data or connected objects

IUnknown
IDoIt
Closure

Parameters

Map IDoIt model to Impl.

18
Composition

• Composites looks like components
• Can test small pieces
• Can also use large ones
• Separate structure from algorithms
• Definition/analysis of relationships
• Eliminates explicit dependencies
• No glue as usually meant
• connection glue is factored out
• algorithmic glue is encapsulated

19
Composition time

• Structure may be decided at
• Design time
• Fully static composition
• Parameterized composition
• Runtime
• Dynamic composition described as changes to
  existing structure
• May be downloaded as add-on software package
• Havent fully explored tradeoffs tending towards
  more runtime composition

20
Example of Composition

• Button to turn on light
• Each piece has one contracts

IPushIt
Button-to-Light Adapter
IButtonEvents
ILight
Button
Light
Desc. Button, Adapterstuff, Light, B-gtA, A-gtL
21
Encapsulation of the Switch

• Edge Connector for lookup/connection

(role)
IPushIt
ILight
Button-to-Light Adapter
IButtonEvents
Button
Light
Desc. Button, Adapterstuff, Light, B-gtA, A-gtL,
EC(out B, in A)
22
Granularity of classes

• Leaf classes
• Up to 1500 lines of C
• Probably less in Java or C
• 1-30 classes per component DLL
• Static composites
• Normal range 5-10 instances
• Packaged like a leaf class
• Connection description very cheap

23
Testing Open Systems

• Late composition
• Usually after component shipped
• Robustness of composite related to pieces
• System testing may be done by the user
• Need to support in-the-field debugging
• Unit testing is critical
• Must strive for more than the 80 case
• Finding evidence of correctness

24
Testing Components

• Verify contract as implemented
• Check pre/post conditions
• Expose contract model variables
• Composites have contracts too
• Verify contracts consumed
• Addresses some integration issues
• Fault injection
• Test and measure the binary form
• Allow feedback to improve contract, impl., etc.

25
What is a robust component?

• Beizer on statement and branch coverage This is
  the weakest criterion in the family testing less
  than this for new software is unconscionable and
  should be criminalized.
• For us
• Contract coverage
• 100 coverage at the object level
• Coverage of all branches in sub-expressions
• Ad hoc approach to selecting paths
• Code review

26
Testing Framework
Pre/Post Checks (delegator)
Contract Scenarios
Test Sequencer
Test Subject
State Mapping Objects
Dependencies
27
Testing a Light
ILight Pre/Post Checks
ILight Scenarios
Test Sequencer
Light
Map ILight model to Impl.
28
Testing a Switch
Switch Pre/Post Checks
Switch Scenarios
Test Sequencer
Switch
Map model to Impl.
ILight Checks
Light
29
Production Process

• One small team per component
• Deliverables include
• Code for one component DLL
• Documentation
• Any new or updated tests
• Different levels of hardness
• Factored, versioned build system
• Audit at end

30
Revisions

• Bugs still happen
• Usually modify and release new component
• In rare cases, release in place

31
Cost of Development

• KLOC
• People Months Classes Ship
  Test
• Composition RT 2 5 5 4 9
• Connection Helper 1 2 1 1 6
• Collections 3 6 6 6 45
• Test framework 1 24 8
• Deployment RT 3.5 26 31 42 63
• Totals 145 43 53 131
  
• KLOC / year shipped 4.4
• KLOC / class 1.2

32
The Road to 100 Coverage
33
Interesting Bugs

• Code and test made same bug
• Coverage of last branch yielded bugs, twice
• Post-condition in test was incomplete
• Several difficult MT bugs
• Shutdown of complex structures

34
Future Work

• Reducing Cost
• Testing Composites
• Increasing Coverage
• Contract Coverage
• Contract Specification

35
Reducing Cost

• Generate pre and post condition delegators
• Involves specification in contract and in
  implementation
• Apply consistency checks to contracts
• Select test cases based on coverage goals
• Testing generics

36
Reducing Cost, cont.

• Test specification language
• Multi-threaded components
• Track low level state and locks held for
  detecting race conditions
• Compiler-generated code
• Reducing noise in output
• Use a GC runtime

37
Testing Compositions

• Static ones are like a component
• Use composition description to help
• Look for new paths not executed by unit test
• Domain-specific
• GUI (composition of bits on screen)
• Network-based (different failure modes)
• Hardware/software composites
• Define contract for composite and try to show
  correspondence with structure

38
Achieving More Coverage

• Loops
• More paths through cleanup code
• Choosing other paths to measure
• E.g., D-U paths in code or in state diagram

39
Contract Coverage

• Sequencing tests
• State walking choosing paths is key
• Genetic algorithm
• Test case correlation
• Track variations generated from specs with actual
  tests

40
Contract Specification Issues

• Reentrancy, events
• Need a description of the state of an instance
  when an event handler is called
• E.g., when button fires events to adapter
• State mutation described in spec
• Can use that to help direct MT testing

41
Related Work

• Brooks, Wirth, Parnas various
• Gall Systemantics
• Petroski Designs and Errors
• Williams Inheritance QI
• Meyer Design by Contract
• Kiczales Open Implementation
• Harel, Wegner Turing revisited

42
Related Work, cont.

• CMU, Nierstrasz Composition
• Gamma, et. al., Coplien Patterns
• DSouza Wills Catalysis
• Beizer, RST Corp Testing
• Beck and others Extreme Prog.
• Holmes, et. all Synchronization
• Szyperski Beyond OOP

43
Summary

• Complex, robust, evolving software systems
• Component and composite approach
• Contracts and testing
• Need better tools!
• http//research.microsoft.com/comapps