Building the Test Environment

1
Building the Test Environment

• The test environment is comprised of all the
  conditions, circumstances, and influences
  surrounding and affecting the testing of
  software. The environment includes the
  organizations policies, procedures, culture,
  attitudes, rewards, test processes, test tools,
  methods for developing and improving test
  processes, managements support of software
  testing, as well as any test labs developed for
  the purpose of testing software and multiple
  operating environments. This category also
  includes assuring the test environment fairly
  represents the production environment to enable
  realistic testing to occur.
• P. 133

2
Four Areas of Focus

• Management Support
• Test Work Processes
• Test Tools
• Testers Competency

3
Management Support

• Without adequate management support testing is
  rarely performed effectively.
• Management tone and Test process alignment are
  the two areas of management support discussed.

4
Management Tone

• Management sets the tone by providing testers the
  resources and management time needed to do their
  job effectively.
• It is representative of the environment that
  management has established that influence the way
  testers work.
• Control environment is part of management tone.

5
Control Environment

• Three things to know about it.
• The control environment is established by the
  highest levels of management and works downward
  through the organization.
• The test function cannot create the
  organizations control environment, but can
  influence how that environment is implemented
  within the test function.
• The control environment will influence the way in
  which testers perform the work which may be
  ethical or unethical.

6
Three sub-areas of Management Tone

• Integrity and Ethical Values
• Commitment to Competence
• Managements Philosophy and Operating Style

7
Integrity and Ethical Values

• An entitys objectives and the way they are
  achieved are based on preferences, value
  judgments and management styles. Those
  preferences and value judgments, which are
  translated into standards of behavior, reflect
  managements integrity and its commitment to
  ethical values.
• Ethical behavior and management integrity are a
  product of the corporate culture. Official
  policies specify what management wants to happen.
  Corporate culture determines what actually
  happens.
• Two areas of Integrity and Ethical values
  incentives temptations and Providing
  Communicating Moral Guidance.

8
Incentives Temptations

• Incentives cited for engaging in fraudulent
  practices and unethical behavior are
• Pressure to meet unrealistic performance targets,
  particularly for short-term results.
• High performance-dependent rewards.
• Temptations for employees to engage in improper
  acts include
• Nonexistant or ineffective controls
• High decentralization of the organization
• A weak internal audit function
• Insignificant penalties for improper behavior
• Removing these incentives temptations can be
  achieved by following sound and profitable
  business practices.

9
Providing and Communicating Moral Guidance

• The most effective way of transmitting a message
  of ethical behavior throughout the organization
  is by example.
• Top management should verbally communicate the
  entitys values and behavioral standards to
  employees.

10
Commitment to Competence

• Competence should reflect the knowledge and
  skills needed to accomplish tasks that define the
  individuals job.
• Management needs to specify the competence levels
  for particular jobs and to translate those levels
  into requisite knowledge and skills.

11
Managements Philosophy and Operating Style

• Managements philosophy and operating style
  affect the way testing is managed, including the
  kinds of business risks accepted.

12
Organizational Structure

• Organizational Structure affects test process
  alignment
• An entitys organizational structure provides the
  framework within which its activities for
  achieving entity-wide objectives are planned,
  executed, controlled and monitored.
• Activities may relate to what is sometimes
  referred to as the value chain inbound
  (requirements) activities, operations or
  production, outbound (software), deployment and
  maintenance.
• An entitys organizational structure depends in
  part on its size.
• An entitys activities will be organized to carry
  out the strategies designed to achieve particular
  objectives.
• Assignment of Authority Responsibility and
  Human Resource Policies and Practices are
  important areas of Organizational Structure.

13
Assignment of Authority and Responsibility

• Management has an important function with the
  assignment of authority and responsibility for
  operating activities, and establishment of
  reporting relationships and authorization
  protocols.
• Alignment of authority and accountability often
  is designed to encourage individual initiatives,
  within limits.

14
Human Resource Policies and Practices

• Human resource practices send messages to
  employees regarding expected levels of integrity,
  ethical behavior and competence.
• These include
• Standards for hiring the most qualified people
• Recruiting practices
• Training policies
• Rotation of personnel and promotions driven by
  periodic performance appraisals demonstrate the
  entitys commitment to the advancement of
  qualified personnel.
• Competitive compensation programs that include
  bonus incentives that serve to motivate
• Disciplinary actions that send a message that
  certain behavior will not be tolerated.

15
Test Work Processes

• Work processes are the policies, standards and
  procedures in a quality IT environment.
• Test work processes are those work processes
  specific to the testing activity.
• Once management commits to create a work
  environment conducive to effective software
  testing, test work processes need to be created.
• It is the testers responsibility to follow these
  work processes and managements responsibility
  if the processes dont work.
• A process to improve the test work processes
  should be implemented to improve the
  effectiveness and efficiency of the policies,
  standards, and procedures.

16
Test Work Processes Contd

• Policy Managerial desires and intents
  concerning either process (intended objectives)
  or products (desired attributes).
• Standards The measure used to evaluate products
  and identify nonconformance. The basis upon which
  adherence to policies is measured.
• Procedure The step-by-step method that ensures
  that standards are met.
• Policies provide direction, standards are the
  rules or measures by which the implemented
  policies are measured, and the procedures are the
  means used to meet or comply with the standards.
• Two test work processes are critical importance
  of work processes, the responsibilities.

17
Test Work Processes Contd

• It is important to a quality IT environment to
  establish, adhere to, and maintain work processes
  in the testing activity.
• Major purposes and advantages of work processes
• Improves communication
• Enables knowledge transfer
• Improves productivity
• Assists with mastering new technology
• Reduces defects and costs

18
Test Work Processes Contd

• It is important that organizations clearly
  establish who is responsible for developing work
  processes (i.e., policies, procedures, and
  standards).
• Policies are need in areas such as
• Building systems
• Testing systems
• Maintaining systems
• Operating systems
• Quality of systems
• Security of systems
• Allocation of resources
• Planning for systems
• Training personnel

19
Test Work Processes Contd

• Key concepts are
• Policies are developed by senior management.
• Policies set direction but do not define specific
  products or procedures.
• Policies are needed in areas that cause problems.
• Policies define the areas in which processes will
  be developed.

20
Test Processes Contd

• The workers who use the procedures and are
  required to comply with the standards should be
  responsible for the development of those
  standards and procedures.
• Management sets the direction and workers define
  that direction.
• Key concepts are
• Management provides the organizational structure
  for workers to develop their own standards and
  procedures.
• The program is driven by management policies.
• Absolute compliance to standards and procedures
  is required.
• A mechanism is provided for the continual
  maintenance of standards and procedures to make
  them more effective.
• The software tester should be the owners of test
  processes and thus involved in the selection,
  development and improvement of test processes.

21
Test Work Processes Contd

• Selecting, developing, and acquiring work
  processes is an overall IT organization
  responsibility.
• Normally the function that performs this activity
  is called the process engineering function.
• Software testers need to both understand how the
  activity operates AND participate when test
  processes, and related processes, are selected
  and put into practice.
• IT groups should develop a plan for implementing
  and operating a process engineering program.

22
Test Work Processes Contd

• Specific components for the process engineering
  program include
• Building a Process Engineering Organization
• Developing a Standard and Procedure for Standards
• Planning for Standards
• Writing, Storing, and Retrieving Standards and
  Procedures.
• Enforcing Standards.

23
Test Work Processes Contd

• The structure that is put into place to develop
  and update policies, standards, and procedures
  must involve both staff and management.
• Guidelines for establishing an organizational
  structure
• Establish a Process Engineering Committee
  comprised of the most senior IT managers.
• Represent all IT organizational areas on the
  Process Engineering Committee
• Appoint an individual as the Process Engineering
  Manager.
• Appoint Ad Hoc Committees to develop individual
  standards and procedures.
• Let the Standards Ad Hoc Committees develop the
  technical standard.

24
Test Work Processes Contd

• Responsibilities of the Process Engineering
  Manager.
• Promote the concept of process engineering
• Be the driving force behind the processes
• Administer the standards program defined by the
  Process Engineering Committee.
• Be a resource to the Process Engineering
  Committee and Ad Hoc Committee.
• Ensure involved parties are adequately trained.

25
Test Work Processes Contd

• Role of the process engineering committee is to
• Accept topics for processes
• Set priority implementation of processes
• Obtain the resources necessary to develop the
  process.
• Approve or reject developed processes.

26
Test Work Processes Contd

• Responsibilities of the Ad Hoc Committee
• Gain representatives from all involved areas.
• Ensure that the committee has between three and
  eight members in size.
• Create the standard and procedure.
• Coordinate reviews of the standard with involved
  parties.
• Periodically review and update the standards and
  procedures previously developed by the Ad Hoc
  Committee.

27
Test Work Processes Contd

• The make up of the process engineering committee
  is important. Some guidelines in selecting
  members are
• Select the highest-level manager who will accept
  the position.
• Assign individuals who are supportive and
  enthusiastic over standards.
• Make long-term assignments to the Process
  Engineering Committee.
• Select individuals who are respected by their
  peers and subordinates.
• Ensure the appropriate areas of interest are
  involved.

28
Test Work Processes Contd

• Guidelines to ensure the appropriate areas of
  interest are included
• Is every organizational function within IT
  involved?
• Are the activities that interface to the IT
  function involved for example, key users and
  customers?
• Are activities having a vested interest involved,
  such as auditors.
• Are all the IT businesses represented such as
  the project leaders, systems programmers, data
  library, security, help desk, and so forth?

29
Test Work Processes Contd

• Prior to creating work processes, the Process
  Engineering Committee must develop a standard and
  procedure for developing standards and
  procedures.
• A standard should define
• A testing policy managements objective for
  testing
• A test workbench to support the policy (i.e.
  standards and processes)

30
Test Work Processes Contd

• Testers workbench input products drive the
  workbench, which uses procedures and standards to
  produce output products.
• The testers workbench is a pictorial
  representation of how a specific test task is
  performed.
• It is the policies that define the objective for
  workbenches.
• A function may need several workbenches, such as
• A budgeting workbench, scheduling workbench, risk
  assessment workbench, and tool selection
  workbench.
• The objective of a workbench is to produce
  defined output products in a defect-free manner.
• It is the execution of the workbench that defines
  worker product quality, and is the basis for
  productivity improvement.

31
Test Work Processes Contd

• Professional test standards.
• Test professionals should be familiar with the
  standards published by organizations such as the
  ISO, U.S. Dept. of Commerce, NIST, and IEEE.
• IEEE standards listed on P. 152

32
Test Work Processes Contd

• Testers are the quality control process for
  developers.
• If high-quality work is needed, quality control
  should be an integral part of the test process.
• Test quality control is performed during the
  execution of the process.
• Analysis is performed after the test process is
  completed.
• Test analysis can only be performed if adequate
  data is maintained during the test process.

33
Test Work Processes Contd

• Eight test analyses common to the test industry.
  P154-158
• Effectiveness and efficiency of test processes.
• The test objectives are applicable, reasonable,
  adequate, feasible, and affordable.
• Testers do not have the needed competencies to
  meet the test objectives.
• The test program meets the test objectives.
• The correct test program is being applied to the
  project.
• The test methodology is used correctly.
• The task work products are adequate to meet the
  test objectives.
• Analysis of the results of testing to determine
  the adequacy of testing.

34
Test Work Processes Contd

• Continuous Improvement process improvement is
  best considered as a continuous process, where
  the organization moves continually around an
  improvement cycle.
• An important step in the improvement cycle is the
  execution of data gathering to establish the
  initial state, and subsequently to confirm
  improvements. A testing process assessment is one
  excellent way to determine the status of your
  current test process.
• Assessment is the thoughtful analysis of
  testing results, and then taking corrective
  action on the identified weakenesses.

35
Test Work Processes Contd

• Testing process assessment is a means of
  capturing information describing the current
  capability of an organizations test process.
• A formal entry to the process occurs wit the
  compilation of the assessment input, which
  defines the
• Purpose why it is being carried out
• Scope which processes are being assessed and
  what constraints apply.
• Any additional information that needs to be
  gathered.
• The input also defines the responsibility for
  carrying out the assessment.

36
Test Work Processes Contd

• Process assessment is undertaken to measure an
  organizations current processes.
• Process assessment is applicable when
• Understanding the state of processes for
  improvement.
• Determining the suitability of processes for a
  particular requirement of class of requirements.
• Determining the suitability of another
  organizations processes for a particular
  contract or class of contracts.
• The framework for process assessment is
• Encourages self-assessment
• Takes into account the context in which the
  assessed process operates.
• Addresses the adequacy of the management of the
  assessed processes through generic practices.
• Is appropriate across all application categories
  and sizes of organization.

37
Test Work Processes Contd

• Continuous improvement
• One of the most commonly identified weaknesses in
  software testing has been the lack of facts
  (metrics), and without facts there is no need to
  take action (improvement).
• Once appropriate measures are identified,
  tracked, and analyzed, then a plan for continuous
  improvement can be implemented.
• The process continues until the appropriate and
  effective improvement mechanisms are uncovered
  and included in the normal process.

38
Test Work Processes Contd

• Eight steps in the Test Process Improvement
  Model.
• Examine the Organizations Needs and Business
  Goals.
• Conduct Assessment
• Initiate Process Improvement
• Analyze Assessment Output and Derive Action Plan.
• Implement Improvements
• Confirm Improvements
• Sustain Improvement Gains
• Monitor Performance

39
Test Work Processes Contd

• In establishing the test environment management
  must assure that the mission/goals of the test
  function are aligned to the mission/goals of the
  organization.

40
Test Work Processes Contd

• Adapting the Test Process to Different Software
  Development Methodologies.
• In the initial days of software development,
  testing was considered a phase of development,
  the testing phase began after software was
  developed costing an estimated 10 times per
  defect compared to a defect found in the
  requirements phase.
• When testing is viewed as a life cycle activity
  testing and development occur together becoming
  an integral part of the development process.

41
Test Work Processes Contd

• Understanding the software development process is
  important for three reasons.
• Understanding the developmental timetable and
  deliverables.
• Integrating the software
• Understanding how software is developed.

42
Test Work Processes Contd

• There are thousands of software development
  methodologies including
• Waterfall
• D-shaped
• Prototype
• Spiral
• RAD
• Incremental
• Book Software Quality Institute described
  strengths and weakenesses.

43
Test Work Processes Contd

• Testers testing software developed by a specific
  software development methodology need to
• Understand the methodology
• Understand the deliverables produced when using
  that methodology which is needed for test
  purposes.
• Identify compatible and incompatible test
  activities associated with the developmental
  methodology.
• Customize the software test methodology to
  effectively test the software base on the
  specific developmental methodology used to build
  the software.

44
Test Work Processes Contd

• In customizing the software testing process to
  the different methodologies the tester needs to
  know
• Will the users be involved in the developmental
  process?
• Do the users understand this developmental
  process?
• Are the users experts in their business
  activity?
• Is this project an enhancement to an existing
  software system developed using another
  methodology?
• Is high reliability an important aspect of this
  software development project and does the
  selected developmental methodology assure high
  reliability?
• Is the amount of changes expected to be
  implemented in the software system consistent
  with the capabilities of developmental
  methodology?

45
Test Tools Contd

• Test Tools Its difficult to perform testing
  economically without the aid of automated test
  tools.
• The most important aspect of software testing is
  the process used to acquire those tools.
• The procedures developed for testers to follow
  during testing should include testing tools and
  techniques.
• Difficulties surrounding the introduction of
  tools can arise in three areas
• Organizational obstacles
• Problems arising from the tools
• Obstacles in the computer environment

46
Test Tools Contd

• The activities associated with the introduction
  of tools should include the following
• Identifying the goals to be met by the tool, and
  assigning responsibility for the activities
  required to meet these goals
• Approving a detailed tool acquisition plan that
  defines the resource requirements for procurement
  and in-house activities
• Approving the procurement of tools and training,
  if this is not explicit in the approval of the
  acquisition plan.
• Determining, after some period of tool use,
  whether the goals have been met.

47
Test Tools Contd

• Test management is assigned responsibility for
• Identifying tool objectives
• Approving the acquisition plan
• Defining selection criteria
• Making the final selection of the tool or the
  source
• The test manager is responsible for the following
  activities
• Identifying candidate tools
• Applying the selection criteria (informal
  procurement) or preparing the RFP (informal
  procurement)
• Preparing a ranked list of tools or sources
• Conducting any detailed evaluations or conference
  room pilots.

48
Test Tools Contd

• The distribution of responsibilities reduces the
  chances of selecting a tool that
• Does not meet the recognized needs of the
  organization.
• Is difficult to use
• Requires excessive computer resources
• Lacks adequate documentation

49
Test Tools Contd

• Event sequencing includes
• employ a methodical process of identifying
  candidate tools and selecting among these based
  on established criteria. This will avoid some of
  the pitfalls of borrowing a tool from an
  acquaintance, etc
• Determine the assignment and training of a Tool
  Manager who can make minor modifications to both
  the computer environment and the tool to provide
  relief from version and release incompatibilities
  with the operating system.

50
Test Tools Contd

• Event Sequence steps P. 171-179
• Goals identified
• Tool objectives desired features and
  requirements
• 2a) Acquisition activities for informal/ formal
  procurment
• e.g. by request or by purchase order
• 3) Procure Tool cost and trial period with
  vendor considerations.
• 4) Evaluation Plan milestones, initial
  performance assessments.
• 5) Implementation Plan responsibilities and
  tasks.
• 6) Training Plan How and Who to train.
• 7) Tool Received
• 8) Acceptance Test test the tool
• 9) Orientation communicate objectives of tool
  use
• 10) Modifications
• 11) Training scheduling and training on use
• 12) Use in the operating environment
• 13) Evaluation Report
• 14) Determine if goals are met

51
Test Tools Contd

• Eight Catagories of test tools
• Automated regression tools
• Defect management tools
• Performance/load testing tools
• Manual tools
• Traceability tools
• Code Coverage
• Test case management tools
• Common tools that are applicable to testing

52
Test Tools Contd

• Most testing organizations agree that if the
  following three guidelines are adhered to tool
  usage will be more effective and efficient.
• Guideline 1 Testers should not be permitted to
  use tools for which they have not received formal
  training.
• The use of test tools should be incorporated into
  test processes so that the use of tools is
  mandatory, not optional.
• Testers should have access to an individual in
  their organization, or the organization that
  developed the tool, to answer questions or
  provide guidance on using the tool.

53
Testers Competancy

• Test competency is the direct responsibility of
  the individual and the organization that employs
  the individual.
• It is based on two criteria
• Skill sets defines the totality of skills that
  need to be learned.
• Performance skills carrying out the skill sets