M

1
MSBased System Development and Testing In a
Net-Centric Environment

• Bernard P. Zeigler, Ph.D., Arizona Center for
  Integrative Modeling and Simulation and Joint
  Interoperability Test CommandFort Huachuca, AZ
  85613-7051zeigler_at_ece.arizona.edu

2
Motivation from a Testing Perspective

• Traditional interoperability concepts and test
  practices are facing severe challenges from
  several sources
• a) complexity within each new system, as well as
  composition into families of systems and systems
  of systems,
• b) extensive use of modeling and simulation in
  simulation-based acquisition, and
• c) the move to service oriented architecture
  (SOA) to support composable services over the
  Global Information Grid (GIG).
• Need a methodology and process to integrate
  development and testing in a net-centric
  environment
• combines with and goes beyond current software
  development paradigms
• rests upon and exploits dynamic systems theory, a
  modeling and simulation (MS) framework, and
  model-continuity development concepts
• Relationship to other software development
  processes will be open for discussion at the end

3
Part 1 Modeling and Simulation Background

• Briefly review the MS framework and theory
• provide background for integrated system
  development and testing process
• overview the Discrete Event Systems Specification
  (DEVS) formalism which provides the computational
  support for the MS framework and theory
• DEVS can serve as the basic medium for
  formalization of standards, analysis of the
  resulting models, automation of the test cases,
  and execution of the test drivers

4
Part 2. MS-Based Development and Testing DEVS
Methodology

• Illustrate MS-Based testing methodology with an
  application drawn from a current effort to
  provide automated test generation for an
  important tactical command and control standard,
  MIL-STD 6016c
• Goal develop a formalized version of the
  MIL-STD 6016C rule sets with the objective
• to complete an unambiguous description of the
  specification,
• enable more automated test development
• Result Automated Test Case Generator (ATC-Gen)
  is a methodology and tool-set based on the DEVS
  formalism.
• Success ATC-Gen will be one of the primary test
  vehicles for the first assessment of an
  integrated air picture system based on MIL-STD
  6016c to occur this fall

5
Framework for MS

• Systems theory-based framework
• provides an ontology for MS that recognizes the
  essential dynamic character of simulation models
• distinguishes the elements in the MS enterprise
  (such as models, simulators, and experimental
  frames) and the relationships (such as validity
  and correctness) that connect such elements in
  meaningful ways related to the objectives of
  simulation exercises,
• provides a rigorous mathematical theory that
  supports manipulations of the elements in their
  real-world incarnations in order to achieve the
  desired relationships

6
MS Entities and Relations
Device for executing model
Real World
Simulator
Data Input/output relation pairs
modeling relation
simulation relation
Each entity is represented as a dynamic
system Each relation is represented by a
homomorphism or other equivalence
Model
structure for generating behavior claimed to
represent real world
7
MS Framework Continuous case
Real World
Simulator
modeling relation
simulation relation

• Numerical Integration
• Accuracy
• Error effects

Model

• Validity
• Accuracy of
• -retro-diction
• -prediction

8
MS Framework Discrete Event case
Real World
Simulator
simulation relation
modeling relation

• Validity
• Accuracy of
• -retro-diction
• -prediction

• Concurrent Processing
• Correctness
• Randomness

Model
9
DEVS within the Framework for MS

• DEVS (Discrete Event Systems Specification)
  formalism
• is a constructive framework for MS
• based on mathematical systems theory
• DEVS enables applications resting on rigorous
  theory
• modeling discrete/continuous heterogeneously
  composed networked systems universality of
  representation
• hierarchical, modular model construction closure
  under coupling
• verifiably correct parallel and distributed
  simulation of ultra-large scale networks
• model-continuity from simulation to execution
  based on abstract simulator
• automated test generation for net-centric
  services based on behavior representation from
  systems theory

10
DEVS Hierarchical Modular Composition Key to
Constructing/Orchestrating Complex Systems

• Atomic lowest level model, contains structural
  dynamics -- model level modularity

Coupled composed of one or more atomic and/or
coupled models
hierarchical construction
coupling
11
Types of Models and their DEVS Representations
Coupled Models
Atomic Models
Partial Differential Equations
Ordinary Differential Equations
Processing/ Queuing/ Coordinating
Networks Collaborations
Physical Space
Phase Based Models
Pulse Based Models (varGen, Sum)
Digraph Models
1,2 Dim Cell Space
Discrete Time/ Automata
Quantum Based Models (DEVS Integrator, instantane
ous Functions
Cellular Automata
can be components in a coupled model
2 Dim State Space
1 Dim State Space
Self Organized Criticality Models
Multi Agent Systems
12
DEVS Theory Nutaro's optimistic simulation
algorithm and its correctness proof

• Nutaro developed a time warp variant that
  correctly simulates all discrete event models
• Previous variants of the time warp algorithm have
  been based on the logical process world view.
• These time warp derivatives have been widely used
  for simulating discrete event models on parallel
  computers,
• But, they can not correctly simulate DEVS
  coupled models with zero-time interactions
• employs a 2-D distributed clock
• A correctness proof for the algorithm was
  constructed
• The correctness proof demonstrates that the
  parallel algorithm simulates exactly the same
  system as the canonical DEVS abstract simulator.
  
• Consequently, parallel and sequential simulation
  of the same DEVS model will always produce the
  same results.
• This is a strong proof of correctness that no
  logical-processor-based proof has been able to
  rival depends strongly on model/simulator
  separation

13
DEVS-Based Model Continuity

• Allows component models of a distributed
  real-time system to be tested incrementally then
  deployed to a distributed environment for
  execution
• Based on replacing DEVS simulator with DEVS Real
  Time executor

• Model continuity is retained between models of
  different design stages
• reduces the occurrence of design discrepancies
  along the development process
• increases the confidence that the final system
  realizes the specification

14
Model Continuity Mixed Virtual/Real Environment
Any number of virtual robots can interact with
a few real robots
15
Summary

• DEVS is an overarching approach
• supports all levels of system interoperability
• using a Systems Theoretic worldview
• component-based characterization of dynamical
  systems in discrete and continuous forms
• methods for modular, hierarchical model
  composition targeting reusability and
  composability

16
Part 2. MS-Based Development and Testing DEVS
Methodology

• DoD acquisition policy requires testing
  throughout the systems development process to
  ensure interoperability and conformance to
  standards.
• The Joint Interoperability Test Command (JITC) is
  striving to improve traditional standards
  conformance and interoperability testing
  methodologies to
• keep pace with the behavioral complexity of new
  systems that are increasingly reliant on advanced
  information technology
• DoD mandated use of MS throughout the
  development life cycle, requires testing methods
  that are themselves MS-based
• In 2003, the JITC started development of a
  formalized version of the MIL-STD 6016C rule
  sets with the objective
• to complete an unambiguous description of the
  specification, and
• enable more automated test development.
• illustrate the methodology with an application
  drawn from a current effort to provide automated
  test generation for an important tactical command
  and control standard, MIL-STD 6016c
• Automated Test Case Generator (ATCGen), which is
  a methodology and tool-set based on the DEVS
  formalism.
• ATC-Gen will be one of the primary test vehicles
  for the first assessment of an integrated air
  picture system based on MIL-STD 6016c to occur
  this fall.
• working towards a proposal for how the DEVS-based
  approach enables
• 1) simulation-based model continuity in
  developing and testing specifications that stem
  from the DoD Architectural Framework (DoDAF), and
• 2) a standard and an infrastructure for
  developing and testing web-based services for
  DoDs emerging Net-Centric Enterprise Services on
  the Global Information Grid.

17
Outline

• JITC responsibility for Link-16 and successors
• Link-16 Challenges to Implementation and Testing
• MSbased Approach to Automated Test Case
• Generation
• Application to Link-16 in the IABM SIAP Context

18
JITC is the Responsible Test Organization for
TDL Standards

• JITC is responsible for ensuring systems that
  implement Tactical Data Link (TDL) (Link
  11/11B/16 and Variable Message Format (VMF)) are
  interoperable and in compliance with the
  applicable joint standards.
• This is accomplished by conducting the following
  types of tests
• Joint / NATO /Combined Interoperability
• Performance Assessment in Operational
  Environments
• Standards Validation
• Standards Conformance
• JITC employs a variety of tools that provide its
  analysts the ability to evaluate TDL system
  performance in both the lab and live
  environments.

source http//jitc.fhu.disa.mil
19
Link-16 Challenges to Implementation and Testing

• Joint Single Link Implementation Requirements
  Specification
• JSLIRS is an evolving standard (MIL-STD-6016c)
  for tactical data link information exchange and
  networked command/control of radar systems
• Presents significant challenges to automated
  conformance testing
• The specification document states requirements in
  natural language
• open to ambiguous interpretations
• The document is voluminous
• many interdependent chapters and appendixes
• labor intensive and prone to error
• potentially incomplete and inconsistent.
• Problem how to ensure that a certification test
  procedure
• is traceable back to specification
• completely covers the requirements
• can be consistently replicated across the
  numerous contexts
• military service, inter-national, and commercial
  companies

20
SIAP/IABM Successor to Link-16

• SIAP (Single Integrated Air Picture) Goal
  Improve the adequacy and fidelity of information
  to form a shared understanding of tactical
  situation
• Integrated Architecture Behavior Model (IABM)
  requires that all sensors utilize a standard
  reference frame for conveying information about
  the location of targets.
• Developed by the Joint SIAP System Engineering
  Organization (JSSEO), Arlington, Va., a
  sub-office of the Assistant Secretary of the Army
  for Acquisition, Logistics and Technology.
• Development costs 160 million over five years
  the individual services will spend an estimated
  600 million
• First major test Config05 next week --
  ATC-Gen will be the basis for testing link-16 and
  extended IABM requirements

http//www.navyleague.org/sea_power/mar_04_18.php
21
Automated Test Case Generator (ATC-Gen)

• IABM is an extension of Link-16 developed in HLA
  environment and requires HLA simulation-based
  testing
• JITC has taken the initiative to integrate
  modeling and simulation into the automation of
  the testing process
• Funded the development of Automated Test Case
  Generator (ATC-Gen) led by ACIMS using DEVS
  (Discrete Event System Specification) technology.
  
• In RD of two years, proved the feasibility and
  the general direction
• The requirements have evolved to a practical
  implementation level, with help from conventional
  testing personnel.

22
Discrete Event Nature of Link-16 Specification
23
Automated Test Case Generation Goals and
Approach

• Goals
• Increase the productivity and effectiveness of
  the conformance testing
• Apply DEVS systems theory, modeling and
  simulation concepts, and current software
  technology to (semi-)automate portions of
  conformance testing

Objective
Automate Testing

Capture Specification as DEVS

Analyze DEVS I/O Behavior
Synthesize Test Models


Test Driver executes models to induce testable
behavior in SUT

Formalized approach for converting standards
documents into test models to run directly
against a system, automating the process to the
extent possible
Interact with SUT over Middleware
24
Benefits of Formalization and Automation

• Provides traceability to original specification
• Reduces ambiguity from textual specification
• Facilitates integrating Modeling Simulation
  into the testing process
• Enables testing of complex
• Standards
• Systems
• Functions
• Families of systems

25
ATC Gen Overall Process

• MIL-STD-6016C is a description of a DEVS that
  mandates the outcome of tests and sequences of
  tests
• ATC Gen analysts convert if-then rules from the
  MIL-STD into XML, defining state variables and
  vectors
• XML if-then rules are used to generate test
  sequences
• Test models are derived from test sequences
• Test Driver runs test models against SUT in
  distributed simulation

26
The Dependency Analyzer (DA) determines the
relationship between rules by identifying shared
state variables Generates test sequences and test
models
Analyst Interpret the text to extract state
variables and rules, where rules are written in
the form If P is true now then do action A
later unless Q occurs in the interim
Test Engineer Develop sets of test
sequences Convert test sequences to executable
simulation models Convert test sequences to
executable simulation models
Test Driver Interacts with and connects to SUT
via HLA or Simple J interfaces Performs
conformance testing of SUTs
27
MIL-STD-6016C Micro-Structure
Message Transmission
Message Phase
Message Preparation
Message Reception

• Stimuli
• Constraints
• Processing

Appendix Structure
Roles
Initiator
Responder
28
Interpretation and Translation
29
Interpretation and Translation
30
Detailed XML Example Traceability, Quality
Assurance

• 4.11.13.12 Execution of the Correlation. The
  following rules apply to the disposition of the
  Dropped TN and the retention of data from the
  Dropped TN upon origination or receipt of a J7.0
  Track Management message, ACT 0 (Drop Track
  Report), for the Dropped TN. The correlation
  shall be deemed to have failed if no J7.0 Track
  Management message, ACT 0 (Drop Track Report),
  is received for the dropped TN after a period of
  60 seconds from the transmission of the
  correlation request and all associated processing
  for the correlation shall be cancelled.
• a. Disposition of the Dropped Track Number
• (2) If own unit has R2 for the Dropped TN, a
  J7.0 Track Management message, ACT 0 (Drop
  Track Report), shall be transmitted for the
  Dropped TN. If the Dropped TN is reported by
  another IU after transmission of the J7.0 Track
  Management message, own unit shall retain the
  dropped TN as a remote track and shall not
  reattempt to correlate the Retained TN and the
  Dropped TN for a period of 60 seconds after
  transmission of the J7.0 Track Management
  message.
• XML Translation
• ltrule trans"4.11.13" stimulus"4.11.13.12"
  reference"4.11.13.12.a.2" ruleName"R2 Unit
  transmits J7.0"gt
• ltcondition txt"Check for R2 own unit"
  expression"AutoCorTrue and (CRair.TNcor.CORtest
  3 and J32.TNref.CORtest3) and
  CRair.R2held1 AND J72.MsgTxTrue"gt
• lt/conditiongt
• ltaction txt"Prepare J7.0 Drop Air Track
  message" expression"J70.TNsrcTNown
  J70.TNrefTNdrop J70.INDex0 J70.INDcu0
  J70.ACTVair0 J70.SZ0 J70.PLAT0 J70.ASchg0
  J70.ACTtrk0 J70.ENV0 MsgTx(J70)"gt
• lt/actiongt
• ltoutput txt"Transmit J7.0" outType"Message"
  outVal"J70"gtlt/outputgt
• lt/rulegt
• ltQAgt
• ltrevisit name"DHF" date"10/16/04"
  status"Open"gtneed to add timer for a period of
  60 seconds in which correlation is not
  reattemptedlt/revisitgt
• lt/QAgt

31
Quality Assurance (QA)

• Provides a history of rule changes
• Identifies possible Change Requests and other
  problematic portions of the standard
• QA tags
• Info Explanation / reasoning for rule
  interpretation
• Revisit Rule needs further analysis set to
  Open or Closed
• Resolution Solution to question / issue

32
MIL-STD-6016C Macro-Structure

• Transactions are atomic units
• Functions are collections of Transactions
• Appendices group related Functions
• Java processing restructures rules into a
  Transaction Module Hierarchy

Appendix N Functional Area, e.g. Track Management
Function P.N e.g. C2 Drop Track
Transaction P.1.N C2 Drop Track Transmit
Atomic Level Unit
Step 1 Stimuli, Preparation, Constraints
Step 2 Processing, Operator Decisions
Step 3 Update Database, Output
33
XML Repository GIG/SOA Asset

• Organized according to MIL-STD-6016C
  macro-structure hierarchy
• RuleCombo folders store aggregations/abstractions
  of lower level rules
• Value added
• Provides a basis for further analysis
• Removes ambiguity
• Annotates problems areas, improving the ability
  to find and fix issues
• Provides organization for indexing states, rules,
  and variables
• Supports test generation and executable rule
  construction

34
Dependency AnalysisVisualization of Rule
Dependencies
Clicking will reveal rule text
Dependencies revealed by hovering
35
Test Sequence Generation Transaction Level Paths
C.1ModuleN active ? infinity
D.1ModuleN active ? infinity
Dependency Between Modules
Potential Test Sequence
U.1ModuleN passiveout1 ? infinity
Desired Sequence C.1 Receive PPLI D.1 Receive
Track U.1 Correlate
36
Determining initial state and message field
values required to drive SUT through sequence

• Analyst
• Determine the data needed to execute a test
  sequence
• Set state variables and field values accordingly

37
Test Simulatlon Architecture
Test Model
System Under Test
J-Series Message Generator
J-Series Message Acceptor
Produces Stimulus
Checks for Proper Response
38
Test Model Construction Translate test
sequences from the DA to derive a composed model
for the Test Driver
39
Test Driver Composition and Deployment
Test Model
Sequence 1
Sequence 2
Sequence 3
Jx1,data1 Jx2,data2 Jx3,data3
Jx1,data1 Jx2,data2 Jx3,data3
Jx1,data1 Jx2,data2 Jx3,data3
Jx4,data4
Jx4,data4
Jx4,data4
Middleware
SUT
40
IABM/Link 16 Correlation Testing

• ATC Gen automated correlation/decorrelation tool
  provides in-depth compliance testing
• Mathematical algorithm of the proximity of the
  tracks
• Logic for prohibitions and constraints
• Test for post-correlation (dropped and retained
  tracks)
• Discriminates between types of failures
  encountered during correlation testing

41
ATC Gen Summary What Can (not) be Automated?
42
Discussion

• Can the DEVS-based MS approach enable
• simulation-based model continuity in developing
  and testing specifications that stem from the DoD
  Architectural Framework (DoDAF), and
• a standard and an infrastructure for developing
  and testing web-based services for DoDs emerging
  Net-Centric Enterprise Services on the Global
  Information Grid.

43
Systems Engineering Perspectives for MS
Applications (Robert Marcus)

DoD System Life-Cycle
44
Interlocking Standards How Do These Play
Together?
Architecture Framework Standards
DODAF
Operational View
Systems View
Technical View
Development Process Standards
Hierarchy of System Specifications
Service Oriented Architecture Standards
Middleware Standards
MS Standards
45
Models and Methodologies How Do These Play
Together?
The MDA defines an approach whereby you can
separate the system functionality specification
from its implementation on any specific
technology platform. That way you can have an
architecture that will be language, vendor and
middleware neutral.
MDA Model Driven Architecture
Petri nets were invented by Carl Adam Petri to
model concurrent systems and the network
protocols used with these systems.
Petri Nets
IDEF Integrated Definition for Function Modeling
UML Unified Modeling Language
MDD Model Driven Development
IDEFØ is a method designed to model the
decisions, actions, and activities of an
organization or system. IDEFØ was derived from a
well-established graphical language, the
Structured Analysis and Design Technique (SADT).
The United States Air Force commissioned the
developers of SADT to develop a function modeling
method for analyzing and communicating the
functional perspective of a system. Effective
IDEFØ models help to organize the analysis of a
system and to promote good communication between
the analyst and the customer. IDEFØ is useful in
establishing the scope of an analysis, especially
for a functional analysis. As a communication
tool, IDEFØ enhances domain expert involvement
and consensus decision-making through simplified
graphical devices. As an analysis tool, IDEFØ
assists the modeler in identifying what functions
are performed, what is needed to perform those
functions, what the current system does right,
and what the current system does wrong. Thus,
IDEFØ models are often created as one of the
first tasks of a system development effort.
"model-driven" development methods, based on
greater use of automation compared to traditional
methods, have already demonstrated their
potential for radical improvements in the quality
of software
This is a 4GL notation tailored to OO software
development. It is primarily a graphical
notation. The standard is managed by the
OMG.integrates the concepts of Booch, OMT and
OOSE by fusing them into a single, common and
widely usable modelling language. UML aims to be
a standard modelling language which can model
concurrent and distributed system
46
Ideal Bifurcated Development Process
Reference Master Model
Simulation based testing in net-centric
environment
Formalization by mapping into DEVS
representations Corresponding levels of System
Specification
Real-time implement-ation and execution
Behavior Requirements at one or more levels of
System Specification
Semi-automated test suite design based on
Experimental Frames
47
Bifurcated Development Process in the DODAF/GIG
Context
Model Design Repository
48
Transfering DEVS-based Testing Methodology to SOA
49
Refining OV-6a (IDEF) to OV-8 (DEVS)

• Inherent problems with IDEF process (,,!),
• Timing, a critical factor in real-time decision
  making
• unaddressed by CPNs, IDEF processes,

IDEF
Automated Code generation of DEVS model thru OV-8
DEVS
LOGIC CODE IN OV-6a (Rules of Engagement/Activati
on of further Activities done with boolean
decision making) IF Significant Movement of
target Then Monitor Target/Target Status
Project Target Movement Target Vector . .
. .. ? Else Monitor for Movement
50
DEVS-based Web-Services Testing
51
Bernard P. Zeigler ACIMSwww.acims.arizona.edu
Contact