Introduction: Skoll: Distributed Continuous Quality Assurance

1
IntroductionSkoll Distributed Continuous
Quality Assurance

• Morimichi Nishigaki

2
Background

• In-house QA
• benefit
• fine level analyses of programs
• shortcomings
• increased QA cost and schedule
• misleading results
• difference from field

3
software trends

• Demand for user-specific customization
• performance-intensive
• Severe cost and time-to-market pressures
• limited resources for highly customizable
  software
• Distributed and evolution-oriented development
  processes
• increase software churn rates, increase the need
  to detect, diagnose and fix faulty changes

New Challenge hundreds of options explosion of
the software configuration space
4
Solution approach

• Skoll
• general, continuous, feedback-driven process
• around-the-world, around-the-clock QA.

Subtask
Subtask
Subtask
QA Process
Subtask
Subtask
Subtask
Subtask
Subtask
Subtask
results
Subtask
Subtask
Central Collection Site
5
Solutions

• Modeling the QA subtask configuration space
• general representation with
• configuration options
• option settings
• inter-option constraints
• temporarily inter-option constrain
• Exploring the configuration space
• search strategy based on uniform sampling
• adaptation strategy for goal-driven process
  adaptation
• Feedback
• automatic characterization, visualization
• Resource availability
• scheduling adapting resource availability
• Process execution framework
• Skoll process provides framework to integrate QA
  techniques and tools

6
Skoll Project

• Developing and evaluating processes, methods, and
  support tools for distributed, continuous QA
• Skoll infrastructure a set of components and
  services
• issues
• how tasks will be decomposed into subtasks
• how will they be implemented so they run on a
  very wide set of client platforms
• how will results be merged together and
  interpreted
• how should the process adapt to incoming results
• how will the results of the overall process be
  summarized

7
Skoll Infrastructure

• Configuration Space Model
• Intelligent Steering Agent
• Adaptation Strategies
• Nearest neighbor
• Temporary constraints
• Terminate/modify subtasks
• Automatic characterization of subtask results
• Visualization
• Subtask Execution

8
Configuration Space Model

• Subtask process parameterized by configuration
  options
• configuration mappings option to a setting
  (Vi,Ci)
• Inter-option constraints limitation of setting
  based on another setting (Pi?Pj)

example
9
Intelligent Steering Agent (ISA)

• Control the global QA process
• find valid configuration to allocate to client
  request
• job configuration package of
• QA subtask implementation
• application code
• configuration parameters
• build instructions
• QA specific code
• Planning problem
• Blockbox planner
• initial state base subtask configuration
• goal state desired configuration, consistent
  with the end user
• output job configuration

10
Adaptation Strategies

• Results returned to ISA is ignored by default
• Monitor, analyze global state to modify future
  subtask assignments
• Nearest neighbor
• test neighbor configurations of failed one
• Temporary constraints
• temporary prevent further selection of job
  configuration
• Terminate/modify subtasks
• monitors for situation where test program provide
  little new information
• switch resources to be spent unexecuted test
  program

example Nearest Neighbor
11
Automatic characterization ofsubtask results

• Used for
• adapting the process
• providing developers
• Classification Tree Analysis (CTA)

example classification of 89 different
configurations
3 categories ERR-1 ERR-2 ERR-3 OK
12
Visualization

• Help developers organize and visualize results
• Server scoreboard manager
• web-based query form

13
Subtask Execution

• Implementing QA subtasks run on client machine
• Server registration manager
• web-based registration form, characterizing
  client platform
• return ID and configuration template to client
• user-specific information, restriction by end
  user
• Skoll client requests job configurations

14
Skoll process
developers
Skoll
query
server scoreboard manager
Automatic Characterization
1
8
configuration model adaptation strategies generic
QA subtask code
Adaptation
7
ISA
registration manager
Skoll client configuration template
job configuration
4
request job configuration
2
5
3
request
6
results
15
Feasibility Study

• ACETAO
• 1MLOC
• run on dozens of OS
• hundreds of options
• distributed 140 developers

16
Study1 Clean Compilation

• Configuration model
• subset of 17 binary-valued compile-time options
• 35 inter-option constraints
• Study execution
• 10 options and 7 constraints 89 valid
  configurations
• random sampling without replacement
• No Nearest neighbor
• 60 of 89 failed
• Lessons
• Iterative model building
• useful automatic characterization

17
Study2 Testing with Default Runtime Options

• Configuration model
• 96 tests
• test-specific options (one option per test )
• Study execution
• 14 compile time options with 13 constrains
• 96 test-specific options with 120 constrains
• 29 configurations (build in Study 1 )
• Results
• compile 98 failed 1979 success
• tests 152 failed 1827 success
• discovered new bugs
• Lessons
• easy extension and refinement
• unreliability of configuration model and tree
  model
• non-deterministic, single observation per valid
  configuration

18
Study3 Testing with Configurable Options

• Configuration model
• 14 compile options with 13 constrains
• 96 test-specific options with 120 constrains
• 6 multi-valued runtime options
• Study execution
• 18792 valid configuration
• 30 min. per test suites, 9,400 hours
• Nearest neighbor
• Results
• scalability
• reconfirm automatic characterization
• adaptation

19
Summary

• Complex subtask configuration spaces are
  interactively modeled.
• Large scale QA processes are developed and
  executed on multiple clients.
• New real bugs in ACETAO were found.
• Automatic failure characterization simplified
  tracking down the causes.

20
END

• Thank you