TQS Teste e Qualidade de Software Software Testing and Quality Software Quality Concepts

1
TQS - Teste e Qualidade de Software(Software
Testing and Quality) Software Quality Concepts
João Pascoal Faria jpf_at_fe.up.pt
www.fe.up.pt/jpf
2
Index

• Software Quality and Testing in the Software
  Engineering Body of Knowledge
• Product and Process
• Notion of Quality
• Quality Attributes
• Quality Costs
• Quality Factors
• Quality Related Activities and Techniques
• Verification Validation versus Quality
  Assurance
• Software Tests, Reviews and Measurements
• Continuous Process Improvement
• Cost-Effective Software Quality

3
Guide to the Software Engineering Body of
Knowledge (SWEBOK)

• Developed by the IEEE Computer Society
• Established with the following objectives
• Promote a consistent view of software engineering
  worldwide.
• Clarify the placeand set the boundaryof
  software engineering with respect to other
  disciplines such as computer science, project
  management, computer engineering, and
  mathematics.
• Characterize the contents of the software
  engineering discipline.
• Provide a topical access to the Software
  Engineering Body of Knowledge.
• Provide a foundation for curriculum development
  and individual certification and licensing
  material.

4
SEWBOK - Knowledge areas (1/2)
5
SEWBOK - Knowledge areas (2/2)
6
SEWBOK - Software Testing
7
SEWBOK - Software Quality
(software testing)
8
Product and process
Software Development Organization (Business
System)
goals
Software Development Process (Business Process)
Demand, Needs
Costumer or Market
Software Product
software developer
software tester
software quality (assurance) engineer
Customer external or internal
resources
Product product or service
Test test and review
Development development and maintenance
9
What is a software product?

• Software product computer programs associated
  documentation
• Software products may be
• Custom (à medida) - developed for a particular
  customer, according to its specifications
• Generic (package) - developed for a general
  market, to be sold to a range of different
  customers
• Types of software products
• Business support software, enterprise software
• Supports the activities of a business or
  organization
• Usually multi-user
• Personal productivity software
• Spreadsheets, word processing tools,
• Usually single-user
• System software
• Operation systems, device drivers, compilers,
  DBMSs, web servers, application servers, etc.
• Embedded software
• Software embedded in devices and machines whose
  main purpose is not generic computation

10
What is a software development process?

• Is the definition of a set of activities whose
  goal is the development or evolution of a
  software product
• Usually comprises the following generic
  activities
• Specification - what the system should do and its
  development constraints
• Development/Implementation - production of the
  software system
• Verification Validation - checking that the
  software meets its specification and is what the
  customer wants (source Ian Sommerville)

Software Development Process
New or changed requirements (problem)
New or changed software product (solution)
(source "Rational Unified Process")
11
The importance of software

• The economies of ALL developed nations are
  dependent on software
• More and more systems are software controlled
• Including an increasing number of safety-critical
  and mission-critical systems, with high demands
  on dependability
• More and more businesses depend on software for
  their success
• Software and Information Systems are critical
  success factors in an increasing number of
  businesses and organizations
• Software engineering expenditure (in the
  development and maintenance of software products)
  represents a significant fraction of GNP (Gross
  National Product) in all developed countries

12
Critical Systems

• Types of critical systems
• Safetycritical system a system whose failure
  may result in injury, loss of life or major
  environment damage
• e.g. an insulin delivery system
• Mission-critical system a system whose failure
  may result in the failure of some goal-directed
  activity
• e.g. a navigational system for a space aircraft
• Business-critical system a system whose failure
  may result in the failure of the business using
  the system
• e.g. a customer account system in a bank
• e.g. the web site of Amazon

13
What is product quality?

• Quality (simplistically) conformance to
  specification / requirements
• This is problematical for software systems,
  because
• Tension between customer quality requirements
  (efficiency, reliability, ...) and developer
  quality requirements (maintainability,
  reusability, ...)
• Specifications are imperfect
• Some quality requirements are difficult to
  specify in an unambiguous way
• Software specifications are usually incomplete
  and often inconsistent
• Quality fitness for purpose
• Quality expectations delivery
• Quality the totality of characteristics of an
  entity that bear on its ability to satisfy stated
  and implied needs

14
Types of quality and customer satisfaction

• Essential quality - attributes necessary to
  achieve minimal levels of customer satisfaction
• Cause dissatisfaction when absent but go
  relatively unnoticed when present because are
  expected or assumed
• Ex customers expect a cordless telephone to
  function in their homes without static and to
  remain charged for a reasonable length of time
• Conventional quality attributes that result in
  satisfaction when present and in dissatisfaction
  when not present
• The-more-the-better the more there is of it, the
  better the customer likes it
• Ex the more years a washing machine operates
  successfully, the better or higher the level of
  customer satisfaction

• Attractive quality - attributes that go beyond
  customer's expectations and desires
• Customers remain satisfied even with the absence
  of these attributes but are delighted with their
  presence
• Ex If a customer brings a car to a garage and
  the mechanic fixes the car at a fair price, the
  customer will be satisfied, because the expected
  service was provided. If the garage also washes
  and vacuums the car, the added service is
  differentiating and may bring the customer
  delight.

(source Karl L. Smart, Assessing quality
documents, 2002)
15
When does a software bug occur?

• The software doesn't do something that the
  product specification says it should do
• The software does something that the product
  specification says it shouldn't do
• The software does something that the product
  specification doesn't mention
• The software doesn't do something that the
  product specification doesn't mention but should
• The software is difficult to understand, hard to
  use, slow, or in the software tester's eyes
  will be viewed by end users as just plain not
  right
• (source Ron Patton, "Software Testing")

16
What is a software error?

• One common definition of a software error is a
  mismatch between the program and its
  specification. DON'T USE THIS DEFINITION. A
  mismatch between the program and its
  specification is an error in the program if and
  only if the specification exists and is correct.
  A program that follows a terrible specification
  perfectly is terrible, not perfect. A better
  definition is A software error is present when
  the program does not do what its end user
  reasonably expected it to do.(source Cem
  Kaner)

17
Bugs, defects, errors, faults and failures

• Structural ...
• Bug (informal) - An unwanted and unintended
  property of a program or piece of hardware,
  especially one that causes it to malfunction.
  Antonym of feature. E.g. "There's a bug in the
  editor it writes things out backward." The
  identification and removal of bugs in a program
  is called "debugging".
• Defect (formal) same as bug.
• Behavioral
• Error - A discrepancy between a computed,
  observed, or measured value or condition and the
  true, specified, or theoretically correct value
  or condition.
• Note some times used as synonymous for bug or
  defect
• Fault (falha) - A manifestation of an error in
  software. A fault, if encountered, may cause a
  failure.
• Failure (falha) - The inability of a system or
  system component to perform a required function
  within specified limits. A failure may be
  produced when a fault is encountered.
• Note Fault tolerant systems have the ability to
  continue normal operation despite the presence of
  hardware or software faults (usually using some
  degree of redundancy) JPF
• (source http//computing-dictionary.thefreedicti
  onary.com)

18
The current status of software quality

• Microsoft Windows XP End-User License Agreement
• 11. LIMITED WARRANTY FOR PRODUCT ACQUIRED IN
  THE US AND CANADA. Microsoft warrants that the
  Product will perform substantially in accordance
  with the accompanying materials for a period of
  ninety days from the date of receipt.() If an
  implied warranty or condition is created by your
  state/jurisdiction and federal or
  state/provincial law prohibits disclaimer of it,
  you also have an implied warranty or condition,
  BUT ONLY AS TO DEFECTS DISCOVERED DURING THE
  PERIOD OF THIS LIMITED WARRANTY (NINETY DAYS).
  ()Some states/jurisdictions do not allow
  limitations on how long an implied warranty or
  condition lasts, so the above limitation may not
  apply to you.()YOUR EXCLUSIVE REMEDY.
  Microsoft's and its suppliers' entire liability
  and your exclusive remedy shall be, at
  Microsoft's option from time to time exercised
  subject to applicable law, (a) return of the
  price paid (if any) for the Product, or (b)
  repair or replacement of the Product, that does
  not meet this Limited Warranty and that is
  returned to Microsoft with a copy of your
  receipt. (..)This Limited Warranty is void if
  failure of the Product has resulted from
  accident, abuse, misapplication, abnormal use or
  a virus.

19
The current status of software quality

• Software is the only product where a high defect
  rate is accepted
• Defect rate in software measured in number of
  bugs per KLOC (1000 Lines Of Code)
• 6-7 defects/KLOC - average defect rate in the USA
  (Jones, C. 2001)
• Increased by 15 in the years 1999-2000, when
  compared to 1997-1998 (Meta Group, January 2002)
• 2.5 crashes/week experienced by frequent users
  (InfoWorld 17/09/2001)
• 10-20 defects/KLOC - typical defect rates in MS
  applications found during unit testing
• 0.5 defects/KLOC - typical defect rates in MS
  applications after release
• 20 defects/KLOC - average defect rate in tested
  code produced by students in work assignment 1 in
  MFES 2005/06

20
Software quality attributes according to the
ISO/IEC 9126 standard

• Distinguishes three views of software product
  quality
• Internal Quality is the totality of
  characteristics of the software product from an
  internal view during its development or
  maintenance
• External Quality is the totality of
  characteristics of the software product from an
  external view during its execution.
• Quality in Use is the users view of the quality
  of the software product when it is used in a
  specific environment and a specific context of
  use. It measures the extent to which users can
  achieve their goals in a particular environment,
  rather than measuring the properties of the
  software itself.
• Ideally, the internal quality determines the
  external quality and external quality determines
  quality in use
• Standards
• ISO/IEC 9126-12001 Software engineering --
  Product quality -- Part 1 Quality model
• ISO/IEC TR 9126-22003 Software engineering --
  Product quality -- Part 2 External metrics
• ISO/IEC TR 9126-32003 Software engineering --
  Product quality -- Part 3 Internal metrics
• ISO/IEC TR 9126-42004 Software engineering --
  Product quality -- Part 4 Quality in use metrics

21
ISSO/IEC 9126 - Quality model framework
22
ISSO/IEC 9126 - Quality model framework
23
ISO 9126-12001- Quality Model for External and
Internal Quality
characteristics
subcharacteristics
24
ISO 9126-12001- Functionality

• Functionality - The capability of the software
  product to provide functions which meet stated
  and implied needs when the software is used under
  specified conditions.
• Suitability - The capability of the software
  product to provide an appropriate set of
  functions for specified tasks and user
  objectives.
• Accuracy - The capability of the software product
  to provide the right or agreed results or effects
  with the needed degree of precision.
• Interoperability - The capability of the software
  product to interact with one or more specified
  systems.
• Security - The capability of the software product
  to protect information and data so that
  unauthorised persons or systems cannot read or
  modify them and authorised persons or systems are
  not denied access to them.
• Functional Compliance - The capability of the
  software product to adhere to standards,
  conventions or regulations in laws and similar
  prescriptions relating to functionality.

25
ISO 9126-12001- Reliability

• Reliability - The capability of the software
  product to maintain a specified level of
  performance when used under specified conditions.
• Maturity - The capability of the software product
  to avoid failure as a result of faults in the
  software.
• Frequency of failure
• Fault tolerance - The capability of the software
  product to maintain a specified level of
  performance in cases of software faults or of
  infringement of its specified interface.
• Recoverability - The capability of the software
  product to re-establish a specified level of
  performance and recover the data directly
  affected in the case of a failure.
• Reliability compliance - The capability of the
  software product to adhere to standards,
  conventions or regulations relating to
  reliability.
• (Availability is considered a combination of
  maturity, fault tolerance and recoverability)

26
ISO 9126-12001- Usability

• Usability - The capability of the software
  product to be understood, learned, used and
  attractive to the user, when used under specified
  conditions.
• Understandability - The capability of the
  software product to enable the user to understand
  whether the software is suitable, and how it can
  be used for particular tasks and conditions of
  use.
• Learnability - The capability of the software
  product to enable the user to learn its
  application.
• Operability - The capability of the software
  product to enable the user to operate and control
  it.
• Attractiveness - The capability of the software
  to be attractive to the user
• Usability compliance - The capability of the
  software product to adhere to standards,
  conventions, style guides or regulations relating
  to usability.
• Note Usually combined with accessibility,
  particularly in the web

27
ISO 9126-12001- Efficiency

• Efficiency - The capability of the software
  product to provide appropriate performance,
  relative to the amount of resources used, under
  stated conditions.
• Time behavior - The capability of the software
  product to provide appropriate response and
  processing times and throughput rates when
  performing its function, under stated conditions.
• Resource utilization - The capability of the
  software product to use appropriate amounts and
  types of resources when the software performs its
  function under stated conditions.
• Efficiency compliance - The capability of the
  software product to adhere to standards or
  conventions relating to efficiency.

28
ISO 9126-12001- Maintainability

• Maintainability - The capability of the software
  product to be modified. Modifications may include
  corrections, improvements or adaptation of the
  software to changes in environment, and in
  requirements and functional specifications.
• Analyzability - The capability of the software
  product to be diagnosed for deficiencies or
  causes of failures in the software, or for the
  parts to be modified to be identified.
• Changeability - The capability of the software
  product to enable a specified modification to be
  implemented.
• Stability - The capability of the software
  product to avoid unexpected effects from
  modifications of the software.
• Testability - The capability of the software
  product to enable modified software to be
  validated.
• Maintainability Compliance - The capability of
  the software product to adhere to standards or
  conventions relating to maintainability.

29
ISO 9126-12001- Portability

• Portability - The capability of the software
  product to be transferred from one environment to
  another.
• Adaptability - The capability of the software
  product to be adapted for different specified
  environments without applying actions or means
  other than those provided for this purpose for
  the software considered.
• Installability - The capability of the software
  product to be installed in a specified
  environment.
• Co-existence - The capability of the software
  product to co-exist with other independent
  software in a common environment sharing common
  resources.
• Replaceability - The capability of the software
  product to be used in place of another specified
  software product for the same purpose in the same
  environment.
• Portability Compliance - The capability of the
  software product to adhere to standards or
  conventions relating to portability.

30
ISO 9126-12001- Quality model for quality in use
31
ISO 9126-12001- Quality model for quality in use

• Characteristics
• Effectiveness
• The capability of the software product to enable
  users to achieve specified goals with accuracy
  and completeness in a specified context of use.
• Productivity
• The capability of the software product to enable
  users to expend appropriate amounts of resources
  in relation to the effectiveness achieved in a
  specified context of use.
• Safety
• The capability of the software product to achieve
  acceptable levels of risk of harm to people,
  business, software, property or the environment
  in a specified context of use.
• Satisfaction
• The capability of the software product to satisfy
  users in a specified context of use.

32
Dimensions of dependability

• Reliability - The probability of failure-free
  system operation over a specified time in a given
  environment for a given purpose
• Availability - The probability that a system, at
  a point in time, will be operational and able to
  deliver the requested services
• Its possibly to have high availability with low
  reliability if failures are repaired quickly
• Safety - The systems ability to operate,
  normally or abnormally, without danger of causing
  human injury or death and without damage to the
  systems environment
• Security The systems ability to protect itself
  from accidental or deliberate external attack
• (source Ian Sommerville)

33
Quality costs

• Costs of conformance
• All costs associated with planning and running
  tests (and revisions) just one time
• Costs of nonconformance
• Costs due to internal failures (before release)
• Cost of isolating, reporting and regression
  testing bugs (found before the product is
  released) to assure that they're fixed (left-hand
  side of fig. 1.2)
• Costs due to external failures (after release)
• If bugs are missed and make it through to the
  customers, the result will be costly product
  support calls, possibly fixing, retesting, and
  releasing the software, and in a worst
  case-scenario a product recall or lawsuits
  (right-hand side of fig. 1.2)
• (source "Software Testing", Ron Patton)

34
Quality costs Costs of nonconformance (1)

• (source "Software Project Survival Guide", Steve
  McConnell)

35
Quality costsCosts of nonconformance (2)

• (source "Software Testing", Ron Patton)

36
Quality costsQuality is free!?

• In his book "Quality is Free The Art of Making
  Quality Certain", Philip Crosby argues that the
  costs of conformance plus the costs of
  nonconformance due to internal failures is
  (usually) less than the costs of nonconformance
  due to external failures
• (source Ron Patton, "Software testing")

costs of conformance
costs of nonconformance due to external failures
lt

costs of nonconformance due to internal failures
37
Quality costsThe optimal amount of testing
(and cost)
(develop, execute and maintain tests)
But how do we know?
Number of bugs found
saturation point gt stop

• (source "Software Testing", Ron Patton)

38
Impacto da falta de qualidade

• Intel gastou 475 m na correcção do erro da
  virgula flutuante do Pentium em 1994 (Computer
  Science, Springer Verlag 1995)
• PrimeCo Personal Communications cancelou contrato
  de 500M com Motorola por causa de falhas (Wall
  Street Journal 24/02/98)
• Time Warner Communications gastou 1B em sistema
  de informação para tentar entrar no negócio
  residencial da rede telefónica (Computerworld
  05/05/97)
• National Bank of Australia perdeu 1,75B devido a
  erro não detectado durante 2 anos (New York Times
  Nov/01)
• Ariane 5 (10 anos de desenvolvimento no valor de
  7B) com uma carga de 500M, explodiu 40 segundos
  após lançamento. Módulo de software gerou evento
  não tratado (ESA 1996)
• Therac-25 ministrou doses incorrectas de Raios X
  em pacientes entre 1985 e 1987 6 mortes (IEEE
  Computer 07/07/93)

39
Costs of software errors

• Direct defect costs (USA)
• Development - 21.2B
• Users - 38.3B (National Institute of Standards
  and Technology, USA, 28/06/2002)
• Consequences cancellations and delays (EUA)
• 293B (Bender, Standish Group 2002)

40
What are the main sources of bugs?

• (source "Software Testing", Ron Patton)

41
Principal product quality factors (1)
Software development process
Budget and Schedule

• (source "Software Engineering", I. Sommervile)

42
Principal product quality factors (2)

• Process quality
• A good process is usually required to produce a
  good product
• For manufactured goods, process is the principal
  quality determinant
• For design-based activity (like software
  development), other factors are also involved
  especially the capabilities of the designers
• For large projects with average capabilities,
  the development process determines product
  quality
• People quality
• For small projects, the capabilities of the
  developers is the main determinant
• Corollary you need lower quality people (and
  higher quality process) in larger projects?
• Project Size x People Quality Constant ?
• Development technology
• Is particularly significant for small projects
• Budget and schedule
• In all projects, if an unrealistic schedule is
  imposed then product quality will suffer

43
Process quality attributes

• (source "Software Engineering", I. Sommervile)

44
Does usage and time cause degradation of a
software product quality?

• By definition, should not, but
• A program running continuously for a long period
  of time may work increasingly slower or even
  crash
• e.g. because of memory leaks or memory
  fragmentation
• may require shutting down and restarting
• do you know products like this?
• Performance decreases with the number of
  concurrent users and the volume of the data
• may require hardware/software upgrade
• Maintainability decreases with time
• may require preventive maintenance (migration to
  new technologies, etc.)
• Software becomes obsolete very quickly
• because of fast evolution of technology,
  requirements or knowledge
• requires continuous innovation and evolution

45
Quality related activities and techniques

• Activities
• Software Verification and Validation (V V)
• Software Quality Assurance (SQA)
• Techniques
• Tests
• Reviews
• Measurements

46
Verification and Validation (V V)

• Goals
• Establish the existence of defects in a product
• Assess whether or not the product is usable in an
  operational situation
• Verification
• Ensure that we are building the product right,
  i.e., according to its specification
• Validation
• Ensure that we are building the right product,
  i.e., according to user needs
• V V are integral part of the development
  process
• Concerned directly with product quality

47
Software Quality Assurance/Management (SQA/SQM)

• Goals
• Ensure that a consistent level of quality is
  achieved in software products, namely, that
  defined standards and procedures are followed.
• Develop a quality culture where quality is seen
  as everyones responsibility.
• Activities
• (Organizationwide) Quality standards definition
• Establish organisational procedures and standards
  for quality. Produce a quality manual
• (Projectwide) Quality planning
• Select applicable procedures and standards for a
  particular project and modify these as required.
  Produce a quality plan.
• (Projectwide) Quality control
• Ensure that procedures and standards are followed
  by the software development team. Produce quality
  review reports
• Quality management should be separate from
  project management to ensure independence of
  budget and schedule pressures
• Concerned directly with process quality and,
  indirectly, product quality

48
Quality Management and Software Development
deliverables

• (source Ian Sommerville)

49
VV versus SQA

• "The goal of a software tester is to find bugs,
  find them as early as possible, and make sure
  that they get fixed"
• "A software quality assurance person's main
  responsibility is to create and enforce standards
  and methods to improve the development process
  and to prevent bugs from ever occurring
• Source Ron Patton

50
VV versus SQA

• SQA governs the procedures meant to build the
  desired quality into the products by assuring
  that the process is well-planned and then applied
  as prescribed
• VV is aimed more directly at product quality,
  in that it is based on testing that can locate
  deviations and fix them. But it also validates
  the intermediate products and therefore the
  intermediate steps of the software engineering
  process
• Source SWEBOK

51
Verification and Validation in the CMMI-SW

• Verification - ensures that the specified
  requirements are satisfied
• Confirms that (intermediate and final) work
  products properly reflect the requirements
  specified for them
• Ensures that you build it right
• Occurs throughout the development of the product
  and work products, beginning with verification of
  the requirements, progressing through the
  verification of the evolving work products, and
  culminating in the verification of the completed
  product
• Validation - ensures that the product will
  fulfill its intended use
• Confirms that the (final) product, as provided,
  will fulfill its intended use
• Ensures that you built the right thing
• Applied mainly to the product and product
  components in its intended environment, but can
  also be applied to work products (e.g.,
  requirements, designs, prototypes) selected on
  the basis of which are the best predictors of how
  well the product and product component will
  satisfy user needs
• Both validation and verification activities often
  run concurrently and may use portions of the same
  environment

52
Quality Assurance in the CMMI-SW

• (Product and Process) Quality Assurance - ensures
  that planned processes are implemented
• Involves objectively evaluating performed
  processes and work products against applicable
  process descriptions, standards, and procedures,
  and communicating and ensuring resolution of
  noncompliance issues
• Usually performed by a QA group that is
  independent of the project, or by peers not
  directly involved in the development or
  maintenance of the work product subject to
  evaluation
• Should begin in the early phases of a project
  (with the participation of the QA group) to
  establish plans, processes, standards, and
  procedures that will add value to the project and
  satisfy its requirements and the organizational
  policies and that will be useable for performing
  QA evaluations, and to designate the specific
  processes and work products that will be
  evaluated
• VV and PPQA may on occasion address the same
  work product but from different perspectives
  care should be taken to minimize duplication of
  effort

53
Main techniques for VV and QA

• Tests
• Dynamic technique, concerned with exercising and
  observing product behaviour to discover defects
• The system is executed with test data (defined
  test cases) and its operational behaviour is
  observed to discover defects (differences between
  observed and expected)
• Used mainly for VV
• GUI testing difficult to automate API testing
  easier to automate
• Reviews
• Static technique - concerned with the analysis of
  the static system representation (source code,
  documentation, ) to discover problems
• May be supplement by tool-based document and code
  analysis
• Measurements
• The value of defined metrics is automatically
  measured on selected components of the product,
  for prediction or control purposes
• Used mainly for QA
• All involve planning, execution and result
  analysis and reporting

54
Typical tests and reviews

• (source "Software Project Survival Guide", Steve
  McConnell)

55
Continuous process improvement

• Conscious quality management comprises continuous
  process improvement
• Process improvement is an integral part of
  business process management
• Several frameworks, standards and reference
  models are concerned with continuous process
  improvement
• ISO 90002000
• Standard establishing the requirements for a
  quality management system that ensures basic
  process capabilities and includes a continuous
  process improvement framework
• Capability Maturity Model Integration (CMMI)
• Model for a staged process improvement from
  maturity level 1 to level 5, and then continuous
  process improvement in maturity level 5
• Probably the most appropriate for software
  development processes
• The PDA Cycle
• Generic framework for the continuous improvement
  of processes, products or systems
• The Six Sigma Initiative
• Generic framework for the continuous improvement
  of processes, products or systems, focusing root
  cause analysis

56
Business Process Management (BPM)

• A software development process is the main
  business process in a software development
  business
• Business process collection of related
  structural activities that produce something of
  value to the organization, its stake holders or
  its customers (http//en.wikipedia.org/wiki/Busine
  ss_Process)
• Business process management set of activities
  which organizations can perform to either
  optimize their business processes or adapt them
  to new organizational needs (http//en.wikipedia.
  org/wiki/Business_Process_Management)
• Business process management comprises
• process modeling and definition
• includes the definition of key performance
  indicators (KPI's)
• process implementation, execution, operation
• process measuring and evaluation
• process improvement and re-engineering

57
The PDCA Cycle

• Originally conceived by Walter Shewhart in
  1930's, and later adopted by W. Edwards Deming
• Is a model that provides a framework for the
  improvement of a process, product or system
• Plan a change or a test, aimed at improvement
• Analyze what you intend to improve, looking for
  areas that hold opportunities for change
• Choose areas that offer the most return for the
  effort you put in
• Do - Carry out the change or test (preferably on
  a small scale)
• Check the results. What was learned? What went
  wrong?
• After you have implemented the change for a short
  time, you must determine how well it is working.
• Is it really leading to improvement in the way
  you had hoped?
• Decide on several measures with which you can
  monitor the level of improvement.
• Act - Adopt the change, abandon it, or run
  through the cycle again
• After planning a change, implementing and then
  monitoring it, you must decide whether it is
  worth continuing that particular change. If it
  consumed too much of your time, was difficult to
  adhere to, or even led to no improvement, you may
  consider aborting the change and planning a new
  one. However, if the change led to a desirable
  improvement or outcome, you may consider
  expanding the trial to a different area, or
  slightly increasing your complexity. This sends
  you back into the Plan phase and can be the
  beginning of the ramp of improvement.
• The PDCA cycle may be applied to all sorts of
  management activities
• project management plan do monitor - control

(source http//www.dartmouth.edu/ogehome/CQI/PDC
A.html )
58
The Six Sigma Initiative

• One of the primary quality initiatives of our
  time, pioneered by Motorola and popularized by
  General Electric
• Six Sigma 6 ? (? standard deviation) 3.4
  defects per million
• In order to reach these results, Six Sigma
  efforts pursue the following five objectives
• Define the problem area in objective terms
• Measure the performance of products and processes
• Analyze the problems to identify root causes
• Improve the results by redesigning processes and
  reducing variation
• Control the processes to ensure the improvements
  are permanent
• (source http//www.proformacorp.com/solutions/six
  sigma.asp )

59
How to achieve high-quality software
cost-effectively?

• Software engineering is concerned with the
  cost-effective development of good software
• How?
• 1st) Defect prevention / avoidance
• 2nd) Defect detection and removal
• (3rd) Fault tolerance)

60
An example Correctness By Construction

• Correctness by Construction (CbyC) is an approach
  that has delivered software with very low defect
  rates cost-effectively.
• The primary goals of very low defect rate and
  very high resilience to change are realized in
  CbyC by two fundamental principles to make it
  difficult to introduce errors in the first place,
  and to detect and remove any errors that do occur
  as early as possible after introduction.
• These principles are achieved by a combination of
  the following six strategies
• Using a sound, formal notation for all
  deliverables. For example, using Z 7 for
  writing specifications so it is impossible to be
  ambiguous, or using SPARK 8 to write the code
  so it is impossible to introduce errors such as
  buffer overflows.
• Using strong, tool-supported methods to validate
  each deliverable. For example, carrying out
  proofs of formal specifications and static
  analysis of code. This is only possible where
  formal notations are used (strategy No. 1).
• Carrying out small steps and validating the
  deliverable from each step. For example,
  developing a software specification as an
  elaboration of the user requirements, and
  checking that it is correct before writing code.
  For example, building the system in small
  increments and checking that each increment
  behaves correctly.
• Saying things only once. For example, by
  producing a software specification that says what
  the software will do and a design that says how
  it will be structured. The design does not repeat
  any information in the specification, and the two
  can be produced in parallel.
• Designing software that is easy to validate. For
  example, writing simple code that directly
  reflects the specification, and testing it using
  tests derived systematically from that
  specification.
• Doing the hard things first. For example, by
  producing early prototypes to test out difficult
  design issues or key user interfaces.

(Source http//www.stsc.hill.af.mil/crosstalk/200
5/12/0512CroxfordChapman.html)
61
An example Correctness By Construction
(Source http//www.stsc.hill.af.mil/crosstalk/200
5/12/0512CroxfordChapman.html)
62
An example Correctness By Construction
SLOC source lines of code
(Source http//www.stsc.hill.af.mil/crosstalk/200
5/12/0512CroxfordChapman.html)
63
References and further reading

• "Practical Software Testing", Ilene Burnstein,
  Springer-Verlag, 2003
• "Software Testing", Ron Patton, SAMS, 2001
• "Software Project Survival Guide", Steve
  McConnell, Microsoft Press, 1998
• "Testing Computer Software", 2nd Edition, Cem
  Kaner, Jack Falk, Hung Nguyen, John Wiley Sons,
  1999
• "Software Engineering", Ian Sommerville,
  6th Edition, Addison-Wesley, 2000 Ian Sommervile
• http//computing-dictionary.thefreedictionary.com
• Correctness by Construction A Manifesto for
  High-Integrity Software, Martin Croxford,
  Roderick Chapman, Praxis High Integrity Systems,
  CrossTalk, The Journal of Defense Software
  Engeneering, Dec 2005, http//www.stsc.hill.af.mil
  /crosstalk/2005/12/0512CroxfordChapman.html
• Capability Maturity Model Integration for
  Software Engineering (CMMI-SW), Software
  Engineering Institute (SEI), Carnegie Mellon
  University (CMU), http//www.sei.cmu.edu/cmmi/
• Guide to the Software Engineering Body of
  Knowledge (SWEBOK), 2004 edition, IEEE Computer
  Society
• ISO/IEC 9126-12001 - Software engineering
  Product quality Part 1 Quality model
• IEEE Std 610.12 1990 - Standard Glossary of
  Software Engineering Terminology