Automated Distributed System Testing

1
Automated Distributed System Testing

• Designing An RTI Verification System

2

• Testing a distributed System ? The Run Time
• Infrastructure (RTI) component of the High Level
• Architecture (HLA)
• Test Strategy
• A test suite had been designed and implemented to
  provide a
• coordinated and automated approach to testing
  this distributed
• system.
• Test Suite Contains
• Creation and application of a Script Definition
  Language (SDL) to specify the test sequences
• Test executive to control execution of the tests.
• Coordinate the test environment
• Record the test results.

3
What is HLA?

• Standard technical architecture for all DoD (US
  Department of Defence) simulations.
• Key Components of HLA include
• A common mechanism for defining and specifying
  object models.
• Services describing an HLA runtime environment
  called HLA interface Specification. It describe
  the functional interface between simulation
  federates and the runtime environments.
• Rules describing policy with the HLA

4
RTI

• The Defense Modeling and Simulation (DMSO)
  sponsored the early development of RTI .
• Services in the specification are documented
  purely from functional perspective,
• No attempt is made to specify the performance ,
  reliability or scalabilty.
• Different implementations of RTI behave
  differently from scaling and other factors point
  of view.
• Testing of RTI focuses on the requirements and
  behavior directly evident in the HLA interface
  Specification.

5
Test Environment Requirements

• Automation of the test execution
• Need for a system to allow automated test
  execution with
• limited intervention.
• Support for repeated multiple executions
• Verification system must support regression
  testing.
• Connectivity to remote processes
• Ability for distributed federates to connect to
  an RTI.
• A coordinated view across all distributed
  processes
• Simulating separate processes and then observing
  the behavior of each process.

6

• Provide a configurable environment
• The verification system should be able to be
  tailored to handle a
• variety of tests, as influenced by the variety of
  HLA services.
• Allow for evolution of the specification
• The verification system must support update of
  the test
• requirements and test specifications to allow
  continued support
• for an evolving HLA specification.
• No Specific hooks into RTI needed
• The verification system should interact with RTI
  via the
• published , code-level, HLA interface without
  modification

7
Test Approaches

• What to Test ?
• Testing an RTI involves
• Invoking a sequence of federate-initiated
  services
• Observing the RTIs response in the form of
  RTI-initiated services (federate callbacks).
• Manual Testing
• Manual testing involves a software Test
  Federate.
• Offers a GUI where an operator can invoke a
  sequence and display the result of RTI service.
• http//hla.dmso.mil ? Get the Test Federate
  software

8

• Using Existing Federations
• Federations created during early developments of
  HLA can be used to test RTIs.
• The constraints of each federation limit the
  sequences of service invocations available to
  test the RTI.
• New Approaches
• Distributed Or Centralized Approach?
• A distributed Verifier would contain a number of
  independent service invocation scenarios or
  scripts.
• To perform a test , the Verifier selects a set of
  scripts , creates an instance of a special
  scripts interpretation federate for each script
  and attaches those to RTI being tested.

9

• A centralized Verifier would also contain a
  number of scripts but it would use only a single
  script to for a given test.
• The script would control all federates involved
  in the test by selectively reaching out to each
  federate and specifying which service to invoke.
• Advantages of Distributed Approach ?
• Scripts can be written in any programming
  language supported by RTI being testes
• Operations can be performed on RTI in parallel.
• Advantages of Centralized Approach?
• Entire content of a test can be expressed in a
  single script.

10
Design
11

• The RTI Verifier system consists of
• SDL created to specify test scripts.
• An Application executive
• Controller
• SDL interpreter to parse and execute scripts
• Test federates to connect and interact with an
  RTI under test
• A data base to maintain requirements, tests and
  test results.

12

• Scripts are created using SDL
• Scripts are stored in database and associated
  with corresponding test requirements.
• A Microsoft Access database application is
  written for entering and maintaining test
  requirements.
• At execution time, a test executive is initiated
  that manage access to test scripts.
• It provide up to 5 separate attachments points
  (APs) to the RTI under test.
• APs connect to test federates.
• Test Federates connected to RTI under test.
• A test controller GUI allows a human tester to
  select , initiate and monitor ongoing test
  activities.
• Scripts stored are retrieved using JDBC.
• Results from scripts execution are also stored in
  database.
• Test results can be viewed online or in a test
  report.

13
Script Definition Language

• Fundamentals of SDL
• Allow invocation of all Federate-initiated
  services
• Allow acceptance of all RTI-initiated services
• Allow interactions with RTI via separate
  attachment points.
• Maintain a consolidated view across all
  attachment points.
• Support HLA data types.
• Provide flow of control
• Provide exit status ( success or failure)

14

• The Basic flow of a SDL script is to
• Initiate one or more RTI service calls
• Look for actions
• Look for lack of actions
• Trap unexpected actions
• Decide what to do next.

15

• SDL Interpreter
• A parser is designed and created to read and
  interpret test statements written in SDL and in
  turn to interact with an RTI via the distributed
  test points.
• JavaCC is a Java language preprocessor that
  generates top-down parsers based upon a detailed
  grammar.
• JJTree is an add-on to JavaCC , allowing to
  generated parser to produce syntax trees.
• Written in Java
• Test Executive
• To execute scripts, the SDL interpreter works
  together with the test executive.
• This process serves as central point of control
  during RTI testing.
• It is responsible for accessing and managing the
  execution of scripts.

16

• A script can be executed either by itself or as a
  part of sequence of scripts that comprise a test.
• Test executive ensures that scripts are executed
  and or aborted as appropriate.
• Test executive records the result in the
  database.
• Written in Java
• Test Controller GUI
• Test controller GUI is graphical front end to the
  test executive that serves as the point of access
  for an RTI tester.
• GUI allows the tester to view the contents of the
  database and to select scripts, tests, series ,
  or scenarios for execution.
• User can monitor the progress of an execution by
  watching the display.
• GUI allows the user to manually invoke any of the
  federate-initiated services and respond to
  RTI-initiated services at any of the attachment
  points.

17

• Test Federates
• A test federate is a completely user-driven
  federate that has the capability to exercise the
  full set of federate-initiated services and
  accept the full set of RTI-initiated services.
• It provides no direct user interface.
• All the running test federates processes are
  controlled completely by test executive.
• Written in Java and use the Java Native Interface
  to call the pre-compiled C RTI libraries.
• Remote Connectivity
• Connectivity between the executive and test
  federate processes is achieved through the Java
  Remote Method Invocation (RMI) protocol.
• The RMI architecture provides a way for objects
  in one java process to invoke methods on objects
  from other java process.
• Test Federate Test Ambassador ?

18

• Database Application and Connectivity
• Connectivity between the test executive and the
  database is achieved with the Java Database
  Connectivity (JDBC).
• JDBC allows the Java programs to execute SQL
  statements on any relational database using a
  single, standard Java interface.
• SQL statements are used to query the database
  (retrieving scripts, tests ,etc)
• To update it ( when recording execution data).

19
Executing Tests

• To test whether an RTI meets a given test
  requirements, the following steps are taken
• The RTI to be tested must be initiated.
• The RTI verifier system must be initiated.
• One or more test federates need to be initiated.
• Once the Verifier system has been initiated, the
  human tester must use the test controller GUI to
  associate test federates.)
• The Human tester must use the test selector GUI
  to select and initiate the particular script,
  test, series, or scenarios to be executed.
• The human tester may monitor the ongoing
  execution of the script , test , series , or
  scenarios and result of this execution to be
  stored in database.

20
(No Transcript)
21
(No Transcript)
22
(No Transcript)