Automatic the regression testing of a mission control system

1
Automatic the regression testing of a mission
control system

• Eduardo Gómez

Hannes Kubr
Oscar González
Gerfried Krames
European Space Agency
Siemens Österreich AG
Deimos Space
2
Background (I)

• regression testing (IEEE)
• Selective retesting of a system or component to
  verify that modifications have not caused
  unintended effects and that the system or
  component still complies with its specified
  requirements
• ESAs ground segment software is subject to
  continuous change
• New releases of the software (e.g. addition of
  functionality)
• Corrections of the software (maintenance)
• Mission specific releases and corrections
• The core functionality does not change, but the
  software needs to be retested every time
  (regresssion testing)
• Testing requires highly specialized personnel
• As new functionality is added, the amount of
  testing increases
• Manual testing procedures are limited by the
  amount of resources available
• Conclusion test automation is a necessity

3
Background (II)

• An ESA study has been conducted to identify the
  core technology
• Contractor Siemens Österreich AG (AT)
• Framework, technology analysis, proof of concept
• The study was later expanded to apply the
  technology to a specific case (on-going)
• Contractor Siemens Österreich AG supported by
  Critical Software (PT) and Terma (DK, DE)
• Based on ESAs mission control system (SCOS-2000)
• Prototype, binding code
• The outputs of the study are being applied to a
  software development project (on-going)
• Deimos Space (ES). Subcontracted by Logica CMG
  (UK)
• ESAs SCOS-2000 forthcoming release

4
HOW? (I)

• Characteristics of ground segment software
• Interacts with end-user based on user interfaces
• Operational Any added code shall not jeopardise
  the reliability
• Many distributed applications involved needs to
  deal with distribution and parallel operation
• Slow evolution, different technologies used
  cannot be bound to a specific technology
• Core asset cannot be bound to proprietary code
  or interfaces
• Formalised based on requirements traceability to
  requirements is needed
• Some specific characteristics of SCOS-2000
  (target of automation)
• Runs on Linux and Solaris
• User interfaces are based on a specific C suite

5
HOW? (II)
Test planning tool (Siemens tool IDATG)
Coordination tool (to be developed)
GUI test driver (Commercial??)
NON-GUI test driver (to be developed)
GUI graphical user interface
6
GUI test drivers (I)

• analog test tool
• Records mouse coordinates and key strokes
• Stops working as soon as geometric parameters
  change (e.g. size)
• Object-based test tool
• Identifies objects in the GUI (does not depend on
  the geometry)
• Strong dependency on tools (no standardize
  language)
• No tool compatible with the system under test
• Embedded code
• Can be activated from an external application
• Very hard to produce
• Potential danger if activated during operations
• Introduces major changes in GUI code

7
GUI test drivers (II)

• Model-view separation into different processes

• Makes the software truly independent from
  visualization package
• The interface can be used for test automation
• - The interface design requires careful planning
  (performance)
• - The view requires independent testing

Test automation system
Model process
View process
CORBA interface

• Implements business logic

- Implements visualization layer
8
language for the driver
Depends on the test interface.
9
The test automation framework (I)
10
The test automation framework (II)

• Separates test preparation from test execution
  (XML interface)
• Moves the complexity to the test driver
• Provides at least two level of abstractions
• High level test flow based on requirement
• Low level Atomic actions based on implementation
  details
• Allows distribution (invocation of remote test
  drivers)
• Provides scalability
• The interface to test drivers is language neutral
  (based on a returned value)
• Communications supported by a simple mechanism
  (test variable service)

11
Test specification and generation tool (I)
Based on Siemens tool IDATG
Different levels of abstraction
Building blocks
Test cases
12
Test specification and generation tool (II)
Low level building blocks encapsulate the access
to the test drives
parameters
Instruction
Instruction details (e.g. name of script)
Support for manual testing
13
The test commander

• Accepts test schedules in the form of XML files
• Test schedule ? test cases? test steps
• Each test step corresponds to a test instructions
  (e.g. invocation of script)
• The test commander allows step by step mode,
  report generation, profiling (including expected
  failures )

Instruction parameters
Test instruction
Test steps
Test results
14
The test drivers and variable service

• Scripts or Programmes that access the test
  interface
• Called by the test commander (normally via shell)
• Return a numerical result
• Open to any language (python is currently the
  preferred choice due to CORBA interfaces)
• Test programmes can be used as test drivers
• Communicate with each other using the test
  variable service
• Based on TCP/IP
• Exchanges variable/value pairs
• Easy to adapt to any language

15
Example
16
Conclusions

• A framework has been presented that allows the
  preparation and automated execution of regression
  tests
• The framework assumes that the business logic of
  the application and the visualization layer run
  on separate processes.
• The framework makes a distinction between test
  preparation and test execution. An XML interface
  has been defined for this purpose.
• The framework does not rely on any specific
  language or technology for the test interface
• The interface complexity is encapsulated under a
  specific piece of code (test driver)
• Test results can be stored in XML format for
  latter post-processing
• The introduction of such a framework encourages
• The reuse of test specifications
• The writing of unambiguous test cases

17
Future work

• Some planned future activities include
• The extension of test automation to other ground
  segment software products
• The analysis of similarities between the
  procedure automation system and the test
  automation system to exploit synergies
• Reuse of code in some cases
• Use of the procedure automation system as a test
  driver
• The analysis of similarities between unit testing
  frameworks such as JUnit and the test automation
  framework with the aim to reuse the code of unit
  test harnesses as automation test drivers