Software Testing: Gets the Bugs out, and More

1
Software Testing Gets the Bugs out, and More

• Dean Cai
• Motorola Computer Group

2
Software Testing in the Life Cycle of a Product

• Life cycle of a telecommunication platform
  product
• Software testing in different stages of a product
  life cycle

3
Challenges of Developing a Telco-grade, High
Availability Platform

• Five-9s and six-9s
• High Availability
• Fault Tolerant
• Open architecture issues
•                         Linux source level
  compatibility
•                        User support in an open
  source environment
• Out-of-box experience
• auto detect and auto configure in an user
  defined environment

4
The Software Design Goals

• Unified implementation running on all platforms
  and multiple operating systems.
• Scaleable software architecture for the
  continuous
• advancement of system availability.
• Ease of integration of telecom applications.
• Ease of integration of telecom specific
  subsystems
• to expand the platform offering.

5
The Software System

• An Architectural view of the software system in a
  complex telecommunication platform
• Example Motorola Computer Groups CPX system

6
A Sample System Application
7
CPX System HA Features

• Hot Swappable CPU and I/O boards
• Dual CompactPCI? backplanes
• Hot Swappable drives, fans, power
• supplies, bridges and alarm panel
• Chassis status control via HSC
• Domain control via HSC
• Telco alarms
• 2N, N1 and Nm configurations

8
Advanced High Availability SW
9
Application Layer

• SNMP v3 Agent
• Monolithic or Master-Subagent models
• Standard MIB2 Support
• UCD extensions for some sysadmin functions
• Motorola MIB for Event, Domain, Alarm and
  Chassis Management
• Graphical User Interface to Event, Domain,
  Alarm and Chassis Management
• Customer provided applications, utilities and
  scripts

10
Application Interface Layer

• APIs (libraries) for event and Alarm Management
• API to general messaging Application interface
• Simplified checkpointing interface
• Database of system component/object definitions
• Policies and Rules for managing system state
  transitions
• Actions and methods/programs for handling state
  transitions
• Utility programs for Event, Domain, Alarm and
  Chassis Management

11
Event Manager Layer

• Clustering agent synchronizes Databases, Rules
  and actions to multiple systems.
• Checkpointing agent provides synchronization
  services for applications and Event Manager.
• System Configuration Event Manager provides
  configurable system
• event and availability management.
• System Alarm manager controls and maintains
  chassis alarms, LEDs,
• Fans and power supplies.

12
Operating System Layer

• PCI services for dynamic insertion/ deletion of
  devices.
• HSC driver for chassis status and control.
• ENUM handler for fielding slot events.
• HA Aware drivers specification and examples to
  include fault detection,
• fault diagnosis, fault recovery and active -
  standby configuration.
• Backplane messaging driver and utilities to
  provide low bandwidth
• board to board communication utilizing the PCI
  bus.
• Event Manager driver funnels event reporting to
  management layer.
• Inter-Host communication for synchronization of
  state and checkpoint data between redundant
  drivers and stacks.

13
Strategies of Testing a Complex Software System
(1)

• Unit test
• Integration test
• Mix of vendor FW, in-house FW, vendor middleware,
  in-house middleware, utility commands, and open
  source OS
• Example pxe driver and hsc driver confliction

14
Strategies of Testing a Complex Software System
(2)

• static int
• pxe_open(struct device dev)
• // __asm__ __volatile__ (".align 2\n\t" ".byte
  241")
• if (net_debug)
• printk("Entered pxe_open()\n")
•  
• if (request_irq(dev-gtirq, pxe_interrupt,
  SA_SHIRQ, "Universal Network Driver", dev))
•  
• /
• if (request_irq(dev-gtirq, pxe_interrupt, 0,
  "Universal Network Driver", dev))
• /
• return -EAGAIN
•  

15
Strategies of Testing a Complex Software System
(3)

• System test
• System level functionalities
• System stress environment
• User friendly evaluation

16
Test Planning (1)

• The process of generating a test plan
• MRD
• SRD
• Design specification
• Test requirement
• Test plan generation
• Review and approval

17
Test Planning (2)

• An example of a test plan

18
Test Planning (3)

• Test scenario design
• Scenario for function coverage
• Scenario for system stress
• Scenario for fault injection, fail-over, and fail
  recovery

19
Test Planning (4)

• Test tool developing
• API test
• Utilities and demo applications using user
  accessible APIs
• Hooks in application and device driver for fault
  injection
• Remote test tools, such as SNMP

20
Test Execution

• DVT (hardware, firmware)
• Software installation and configuration
• Utility command based testing (feature
  verification, code validation)
• Stress test
• Expose potential timing issues (HW and SW)
• Expose potential system resource allocation
  problems
• Memory leak
• User application simulation (user scenario test,
  API validation)

21
Problem Reporting, Resolving, and Verifying

• Quality control from software testing perspective
• The process flow of our in-house problem
  reporting system (CPR)

22
Test Management and Automation

• Ways to improve the quality of testing
• Process control
• Automation
• An sample approach
• System Test Management Framework

23
Project Goals

• Increase test coverage
• Increase test efficiency
• Reduce System Test Cycle Time
• Reduce preparation time and learning curve
• Increase test traceability
• Built-in process control

24
Description of the Project (1)

• Identify requirements for test coverage
• Analyze current test coverage
• Examine current test tools
• Identify the needs for improving existing tools
• Identify the needs for new tools

25
Description of the Project (2)

• Improve the existing tools
• Develop new tools
• Select a test framework package (TETware, HP-VUE,
  LabView, etc.)
• Consolidate tools into an user-friendly framework
• Develop an user front-end of the framework for
  test configuration and deployment

26
Description of the Project (3)

• Build a test database for collecting and
  archiving test data
• develop a test plan generator
• develop a test report generator
• Develop a CM package for tool development and
  maintenance
• Staff training

27
Analyze Current Test Coverage

• Overall system coverage
• DVT coverage (CPU, backplane, IO slots, alarm
  panels, power supply, etc.)
• Software functionality coverage
• Software integration coverage
• System functionality coverage
• System performance coverage

28
Overall System Coverage
No-needs
GoAhead Test Suite
Manual test
AIX Suite ports
Florian
HCT
Overlaps
HA tests
Holes
100 coverage
29
Software Functionality Coverage
Unit Test
Bd Level DVT
Config. X
OS Integ Test
Config. 2
System Test
Config. 1
OS
Appl
Driver
API
30
Test Framework Architecture
User Interface
Test Plan Generator
Test Event Log
User Login
Report Generator
Test Doc Database
HW Configurator
HW Config Database
SW Configurator
Data Collector
Test Data Database
Tool Download/ Rm mount
SW Config Database
Go
Test Tool Studio
Test Monitor
Test Tool Lib
31
Sample Test Flow (1)
Platform Configurator
Suite 1
Test Suite Configurator
Platform
CPX8221
CPX1205
Suite 2
Gemini 2.0
Suite X
1. Login on Test Server 2. Select Platform 3.
Enter platform config data 4. Select test
suite(s) to run 5. Enter test suite config data
32
Sample Test Flow (2)
Server
Server
Go
Target hostname
TH-1
Test suite
Test data
TH-2
Target
Target
TH-X
6. Enter target host name 7. Test suite
download from Server to Target 8. Target
console opens on the server login 9. Test suite
starts on Target 10. Test data transfer from
Target to Server
33
Test Framework Selection

• TETware - Unix Script type, no GUI
• LabView
• HP-VUE
• VisualBASIC
• SNMP Manager