CSTE Common Book of Knowledge Version 6'2

1
CSTECommon Book of KnowledgeVersion 6.2

• Skill Category 1 (Part 1)Software Testing
  Principles and Concepts

Prepared by Mike Webb for the SISQA sponsored
CSTE Study Group
2
Preview of Software Testing and Principles

• 1.1 Vocabulary
• 1.2 What is Life Cycle Testing?
• 1.3 Reducing the Frequency of Defects in
  Software Development
• 1.4 Factors Affecting Software Testing
• 1.5 Life Cycle Testing

3
Testing is aQuality Control Activity

• There is often confusion regarding the difference
  between quality control (QC) and quality
  assurance (QA).
• Quality methods can be segmented into two
  categories preventive methods and detective
  methods.
• Quality has two working definitions
• Producers Viewpoint The quality of the product
  meets the requirements.
• Customers Viewpoint The quality of the product
  is fit for use or meets the customers needs.
• Both quality assurance and quality control are
  necessary.

1.1.1 page 1-2
4
Quality Assurance (QA) vs. Quality Control (QC)

• Quality Assurance
• a planned and systematic set of activities
  necessary to provide adequate confidence that
  products and services will conform to specified
  requirements and meet user needs. Quality
  assurance is a staff function.
• Quality Control
• the process by which product quality is compared
  with applicable standards, and the action taken
  when nonconformance is detected. Quality control
  is a line function.

1.1.1.1 p 1-2 1.1.1.2 p 1-3
5
Quality Control (QC)

• QC relates to a specific product or service.
• QC verifies whether specific attributes are in,
  or are not in, a specific product or service.
• QC identifies defects for the primary purpose of
  correcting defects.
• QC is the responsibility of the team/worker.
• QC is concerned with a specific product.

1.1.1.2 p 1-3
6
Quality Assurance (QA)

• QA helps establish processes.
• QA sets up measurement programs to evaluate
  processes.
• QA identifies weaknesses in processes and
  improves them.
• QA is a management responsibility, frequently
  performed by a staff function.
• QA is concerned with all of the products that
  will ever be produced by a process.
• QA is sometimes called QC over QC because it
  evaluates whether QC control is working.
• QA personnel should never perform QC unless it is
  to validate QC.

1.1.1.2 p 1-4
7
The Cost of Quality

• The Cost of Quality is all the costs that occur
  beyond the cost of producing the product right
  the first time.
• Used to quantify the total cost of prevention and
  appraisal, and costs associated with the
  production of software.
• Includes all costs associated with the
  prevention, identification, and correction of
  product defects.
• (See Figure 1.1 on next slide)

1.1.2 p 1-4
8
The Cost of Quality (Fig 1-1)
1.1.2 p 1-4
9
Costs Associated with Producing Quality Products

• Prevention Costs
• Money required to prevent errors and to do the
  job right the first time. Normally requires
  up-front costs for benefits that will be derived
  months or even years later. Includes money spent
  on establishing methods and procedures, training
  workers, acquiring tools, and planning for
  quality. Prevention money is all spent before the
  product is actually built.
• Appraisal Costs
• Money spent to review completed products against
  requirements. Includes the cost of inspections,
  testing, and reviews. Appraisal money is spent
  after the product is built but before it is
  shipped to the user or moved into production.
• Failure Costs
• All costs associated with defective products that
  have been delivered to the user or moved into
  production. Some failure costs involve repairing
  products to make them meet requirements. Others
  are costs generated by failures such as the cost
  of operating faulty products, damage incurred by
  using them, and the costs associated with
  operating a Help Desk.

1.1.2 p 1-5
10
Cost of Quality (cont.)

• The Cost of Quality varies from one organization
  to the next.
• Majority of the Cost of Quality is associated
  with identification and correction of defects.
• To minimize production costs, the project team
  must focus on defect prevention.
• The goal is to optimize the production process
  such that rework is eliminated and inspection is
  built into the production process.
• Applying the concepts of continuous testing to
  the systems development process can reduce the
  Cost of Quality.

1.1.2 p 1-5
11
Software Quality Factors (SQF)

• These are attributes of the software that, if
  they are wanted and not present, pose a risk to
  the success of the software, and thus constitute
  a business risk.
• The primary purpose of applying SQF in a software
  development program is to improve the quality of
  the software product.
• Eleven SQF were identified by McCall in 1977.
  (See the Software Quality Factor Survey form in
  Figure 1-3 on the next slide.)

1.1.3 p 1-5
12
Software Quality Factors Survey Form
(Figure 1-3)
1.1.3.1 p 1-7
13
How Quality is Defined

• There are multiple quality philosophies, but most
  contain the same core components
• Quality is based upon customer satisfaction
• Your organization must define quality before it
  can be achieved
• Management must lead the organization through any
  improvement efforts
• Peter R. Scholtes introduces the contrast between
  effectiveness (doing the right things) and
  efficiency (doing things right). Quality
  organizations must be both effective and
  efficient.

1.1.4 p 1-12
14
Five Perspectives of Quality

• Each of these perspectives must be considered as
  important to the customer as the others
• Transcendent I know it when I see it
• Product-Based Possesses desired features
• User-Based Fitness for use
• Development- and Manufacturing-Based
  Conforms to requirements
• Value-Based At an acceptable cost

1.1.4 p 1-12
15
Quality in Fact and Quality in Perception

• Patrick Townsend examines quality in fact and
  quality in perception as shown in Table 1-3.
• Quality in fact is usually the supplier's point
  of view, while quality in perception is the
  customer's.

1.1.4 p 1-12
16
Definitions of Quality

• Quality is meeting the customer's requirements
  the first time and every time.
• Quality is conformance to a set of customer
  requirements that, if met, result in a product
  that is fit for its intended use.
• Quality is much more than the absence of defects.
• Quality can only be seen through the eyes of the
  customers.
• Exceeding customer expectations assures meeting
  all the definitions of quality.

1.1.5 p 1-13
17
What is Quality Software?

• The producers view of quality software means
  meeting requirements.
• The customers/users view of software quality
  software means fit for use.
• These two definitions are not inconsistent.
• Meeting requirements is the producers definition
  of quality it means that the person building the
  software builds it in accordance with
  requirements.
• The fit for use definition is a users definition
  of software quality it means that the software
  developed by the producer meets the users need
  regardless of the software requirements.

1.1.6 p 1-13
18
The Software Quality Gap
Figure 1-6 The Two Software Quality Gaps
1.1.6.1 p 1-14
19
What is Life Cycle Testing?Why Do We Test
Software?

• The simple answer as to why we test software is
  that developers are unable to build defect-free
  software.
• If the development processes were perfect,
  meaning no defects were produced, testing would
  not be necessary.
• Testing is the process of evaluating a
  deliverable with the intent of finding errors.

1.2.1 p 1-15
20
Developers are not Good TestersHere are some
reasons

• Misunderstandings will not be detected, because
  the checker will assume that what the other
  individual heard from him was correct.
• Improper use of the development process may not
  be detected because the individual may not
  understand the process.
• The individual may be blinded into accepting
  erroneous system specifications and coding
  because he falls into the same trap during
  testing that led to the introduction of the
  defect in the first place.
• Information services people are optimistic in
  their ability to do defect-free work and thus
  sometimes underestimate the need for extensive
  testing.

1.2.2 p 1-16
21
What is a Defect?

• A defect is an undesirable state.
• Two types of defects process and procedure.
• The term quality is used to define a desirable
  state. A defect is defined as the lack of that
  desirable state. In order to fully understand
  what a defect is we must understand quality.

1.2.3 p 1-16
22
Software Process Defects

• Ideally, the software development process should
  produce the same results each time the process is
  executed.
• Variability is the enemy of quality
• The concept of measuring and reducing variability
  is commonly called statistical process control
  (SPC).
• Testers need to understand process variability,
  because the more variance in the process the
  greater the need for software testing.

1.2.4 p 1-17
23
What Does It Mean for a Processto be In or Out
of Control?

• A process is defined as stable if its parameters
  (i.e., mean and standard deviation) remain
  constant over time it is then said to be in a
  state of statistical control. Such a process is
  predictable.
• Special causes of variation are not typically
  present in the process. They occur because of
  special or unique circumstances. If they exist,
  the process is unstable or unpredictable. Special
  causes must be eliminated to bring a process into
  a state of statistical control.

1.2.4.1 p 1-17 though p 1-21
24
In Control Out of Control Processes
Figure 1-7An In-Control (Stable) Processp 1-18
Figure 1-8An Out-of-Control(Unstable) Processp
1-20
1.2.4.1 p 1-17 though p 1-21
25
Examples of Software Design Defects

• Designing software with incomplete or erroneous
  decision-making criteria. Actions have been
  incorrect because the decision-making logic
  omitted factors that should have been included.
• Failing to program the software as intended by
  the customer (user), or designer, resulting in
  logic errors often referred to as programming
  errors.
• Omitting needed edit checks for determining
  completeness of output data. Critical data
  elements have been left blank on many input
  documents, and because no checks were included,
  the applications processed the transactions with
  incomplete data.

1.2.5.1 p 1-23
26
Examples of Data Defects

• Incomplete data used by automated decision-making
  applications. Some input documents prepared by
  people omitted entries in data elements that were
  critical to the application but were processed
  anyway. The documents were not rejected when
  incomplete data was being used.
• Incorrect data used in automated decision-making
  application processing. People unintentionally
  introduced incorrect data into the IT system.
• Obsolete data used in automated decision-making
  application processing, perhaps due to new
  circumstances, or the new data was available but
  was not put into the computer.

1.2.5.2 p 1-23
27
Finding Defects

• All testing focuses on discovering and
  eliminating defects or variances from what is
  expected.
• Testers need to identify these two types of
  defects
• Variance from Specifications A defect from the
  perspective of the builder of the product.
• Variance from what is Desired A defect from a
  user (or customer) perspective.

1.2.6 p 1-24
28
Typical Software System Defects

• Improperly interpreted requirements
• Users specify the wrong requirements
• Requirements are incorrectly recorded
• Design specifications are incorrect
• Program specifications are incorrect
• Errors in program coding
• Data entry errors
• Testing errors
• Corrected condition causes another defect

1.2.6 p 1-24
29
Reducing the Frequency of Defectsin Software
Development

• Department of Defense (DOD) undertook a study to
  identify the criteria that made software
  development more effective.
• Research was conducted by Carnegie Mellon
  Universitys Software Engineering Institute
  (SEI).
• The end result was an algorithm that enabled the
  DOD to identify the more effective and efficient
  software development. This algorithm is now known
  as the Capability Maturity Model.

1.3 p 1-25
30
The Five Levels of Maturity

• The capability maturity model identifies five
  different levels of maturity. As the model moves
  from Level 1 to Level 5, the process variability
  is significantly reduced. Thus, those at Level 5
  have minimal variability in their software
  development process, while Level 1 organizations
  have significant variability.
• The cost differences to produce a function point
  of logic between a Level 1 and Level 5
  organization may vary by 100 times. In other
  words, what a Level 1 organization may spend on
  building software, for example 1,000, may only
  cost 10 for a Level 5 organization.

1.3.1 p 1-25
31
The Five Levels of Process Maturity
Review the details on pages 1-26 through 1-28
Figure 1-10
1.3.1 p 1-25
32
The Five Levels of Process Maturity

• Level 1 Ad Hoc
• Level 2 Control
• Level 3 Core Competency
• Level 4 Predictable
• Level 5 Innovative

1.3.1 p 1-25
33
Factors Affecting Software Testing

• People relationships
• Scope of testing
• Misunderstanding life cycle testing
• Poorly developed test plans
• Testing constraints

1.4 p 1-29
34
People Relationships

• The top ten people challenges have been
  identified as
• Training in testing
• Relationship building with developers
• Using tools
• Getting managers to understand testing
• Communicating with users about testing
• Making the necessary time for testing
• Testing over the wall software
• Trying to hit a moving target
• Fighting a lose-lose situation
• Having to say no
• According to the book Surviving the Top Ten
  Challenges of Software Testing, A People-Oriented
  Approach by William Perry and Randall Rice

1.4.1 p 1-29
35
Scope of Testing

• Among the broader scope of software testing are
  these responsibilities
• Software testing can compensate for the fact that
  the software development process does not
  identify the true needs of the user, and thus
  test to determine whether or not the users needs
  have been met regardless of the specifications.
• Finding defects early in the software development
  process when they can be corrected at
  significantly less cost than detected later in
  the software development process.
• Removing defects of all types prior to software
  going into a production state when it is
  significantly cheaper than during operation.
• Identifying weaknesses in the software
  development process so that those processes can
  be improved and thus mature the software
  development process.

1.4.2 p 1-30
36
Misunderstanding Life Cycle Testing

• The traditional view of the development life
  cycle places testing prior to operation and
  maintenance. See Table 1-5 on next slide
• All too often, testing after coding is the only
  method used to determine the adequacy of the
  system. When testing is constrained to a single
  phase and confined to the later stages of
  development, severe consequences can develop.
• It is not unusual to hear of testing consuming 50
  percent of the development budget. All errors are
  costly, but the later in the life cycle that the
  error discovered is made, the more costly the
  error.

1.4.3 p 1-31
37
Life Cycle Testing Activities
Table 1-5
1.4.3 p 1-32
38
Life Cycle Testing Activities (cont.)

• Studies show the majority of system errors occur
  in the design phase. Approximately two-thirds of
  all detected system errors can be attributed to
  errors made during the design phase.
• The recommended test process involves testing in
  every phase of the life cycle.
• Requirements phase emphasis is on validation to
  determine that the defined requirements meet the
  needs of the organization.
• Design and program phases emphasis is on
  verification to ensure that the design and
  programs accomplish the defined requirements.
• Test and installation phases emphasis is on
  inspection to determine that the implemented
  system meets the system specification.
• Maintenance phases the system is retested to
  determine that the changes work and that the
  unchanged portion continues to work.

1.4.3 p 1-32
39
Life Cycle Testing Activities (cont.)

• Study sections 1.4.3.1 through 1.4.3.6 starting
  on page 1-33 for details about the testing
  process for each phase of the life cycle.
• Requirements
• Design
• Program (Build/Construction)
• Test Process
• Installation
• Maintenance

1.4.3.1 p 1-33 through 1.4.3.6 p 1-34
40
Poorly Designed Test Planning

• Variability in test planning is a major factor
  affecting software testing.
• A plan should be developed that defines how
  testing should be performed.
• Four steps must be followed to develop a
  customized test plan
• Select and rank test objectives.
• Identify the system development phases.
• Identify the business risks associated with the
  system under development.
• Place risks in the matrix.

1.4.4 p 1-35
41
Testing Constraints

• Anything that inhibits the testers ability to
  fulfill their responsibilities is a constraint.
• Constraints include
• Limited schedule and budget
• Lacking or poorly written requirements
• Changes in technology
• Limited tester skills

1.4.5 p 1-36
42
Budget and Schedule Constraints

• The cost of defect identification and correction
  increases exponentially as the project
  progresses.
• A defect encountered during requirement and
  design is the cheapest to fix.
• Assume that it costs x to fix a defect in the
  requirement and design phase. Fixing that same
  defect during the system test phase costs 10x. A
  defect corrected after the system goes into
  production costs 100x.

1.4.5 .1 p 1-36
43
Relative Cost versus the Project Phase
Figure 1-12
1.4.5 .1 p 1-37
44
Testing Cost Curve
Figure 1-13
1.4.5 .1 p 1-38
45
Lacking or Poorly Written Requirements

• Test objectives enable the test manager and
  project manager to gauge testing progress and
  success.
• Each test objective should contain a statement of
  the objective, and a high-level description of
  the expected results stated in measurable terms.
  The users and project team must prioritize the
  test objectives.
• Usually the highest priority is assigned to
  objectives that validate high priority or
  high-risk requirements defined for the project.
• In cases where test time is cut short, test cases
  supporting the highest priority objectives would
  be executed first.
• Test objectives can be easily derived from using
  the system requirements documentation, the test
  strategy, the outcome of the risk assessment, and
  the test team assignments.

1.4.5 .2 p 1-38
46
Changes in Technology

• Technological developments that can cause
  organizations to revise their testing approach
• Integration
• System Chains
• The Domino Effect
• Reliance on Electronic Evidence
• Multiple Users

1.4.5 .3 p 1-39
47
Limited Tester Skills

• Testers should be competent in all ten of the
  CSTE Common Body of Knowledge skill categories to
  be effective.
• Lack of the skills needed for a specific test
  assignment constrains the ability of the testers
  to effectively complete that assignment.

1.4.5 .3 p 1-39
48
Life Cycle Testing

• Life cycle testing involves continuous testing of
  the solution even after software plans are
  complete and the tested system is implemented.
• Life cycle testing cannot occur until you
  formally develop your process
• Life cycle testing is best accomplished by
  forming a test team. The team is composed of
  project members responsible for testing the
  system. They must use structured methodologies
  when testing they should not use the same
  methodology for testing that they used for
  developing the system.

1.5 p 1-40
49
Figure 1-14Life Cycle Testing
1.5 p 1-41
50
Creating S.M.A.R.T. Goals

• Specific
• Measurable
• Attainable
• Realistic
• Timely

(not CBOK related general info)