Status Update of NATO TG016

1
Status Update of NATO TG-016
2
Agenda

• Introduction
• Motivation
• Document Status
• Scope of Document
• Overview of Content
• Schedule forward
• Issues
• Summary

3
Introduction
The primary objective of the NATO TG-016 is to
provide the NATO community with guidance related
to the verification and validation (and
accreditation, where needed) of
federations. Members Daniel Girardot France
Ingo Cegla Germany Sigfried Pohl
Germany Dirk Brade Sweden Mark Dumble
UK Simone Youngblood US
4
Motivation
The NATO/PfP community currently has several
major initiatives that require the development of
an MS federation (e.g., PATHFINDER, NIREUS,
etc). The success of these initiatives is tied to
the credibility of the federation results and
therefore, the VVA process. This makes
definition of a consistent VVA process even more
imperative.
5
Document Status
Initial Draft October 02 Alpha Review to PDG
for Comment March 04 Comments from PDG March
04 TG-016 Review of Comments April
04 Development of Generic Process Model May
04 TG-016 Revision based on Comments June 04 and
Generic Process Model
6
Scope of Document

• Generally speaking, the objectives aim
• To assess technical interoperability within the
  HLA federation,
• To assess substantive interoperability among the
  federates,
• to establish traceability between the
  intermediate products of each phase of the FEDEP
• To assess internal consistency of the products in
  the FEDEP.
• VV of Federations
• This document provides guidance on VV during
  each step of the HLA FEDEP (IEEE 1516.3) to
  support the substantive and technical
  interoperability of federates in an HLA
  federation. The document focuses on the
  composition of models as a distributed simulation
  within the technical framework defined by the
  HLA.
• VV of Federates
• The VV of individual federates is outside the
  scope of this document. It is assumed that for
  each federate both a VV history and a conceptual
  model are available.
• Accreditation
• Throughout this document, accreditation is
  addressed as a distinct process, which drives
  VV. It thereby influences VV in all phases of
  the FEDEP. The planning and implementation of VV
  activities focuses on the creation of VV
  evidence that supports the accreditation
  decision. Whenever VV evidence is created it
  should be passed as soon as possible to those
  responsible for accreditation (accreditation
  agent or accreditation authority, as applicable),
  who assess the evidence and provide feedback to
  the VV agent.
• Intended audience of this document
• This document is intended to be used by those
  planning and implementing VV of an HLA
  federation, as well as those who have been tasked
  to make or support an accreditation decision
  (e.g. accreditation agent). It is assumed that
  the reader is familiar with the basic concepts of
  the HLA and possesses at least basic VV
  knowledge.
• Further information is identified in the
  recommended reading list in the Bibliography.

7
Overview of Content
VV
assessment
statement
Residual
Define
AC
uncertainty
Identify
acceptable
Create ToA and determine acceptable residual
uncertainty
determination
residual
uncertainty
ToA
Get overview over available model information and
available real system knowledge
Acquire
model
and system
information
Model
information and
VV
report
Plan VV and Derive ToVV (Decomposition)
system
knowledge
Assess VV (Re-composition)
Review
Plan VV
conducted
VV
ToVV
Assessed
items
of
evidence
Implement VV
Assess Evidence
Implement
Review
VV
Evidence
Items
of
evidence
8
Overview of Content

• 1. Define Acceptance Criteria Identify
  acceptable residual uncertainty
• (Focus on suitability correctness)
• Review problem (identify key parameters) Note
  dependency with Federation Objective statements
• Define Measures of Acceptance
• Quantification of Acceptability
• Impact Analysis (Risk Assessment)
• Evolve for more completeness (at all phases)
• Minimise
• Refine

9
Overview of Content

• 2. Acquire Model and Simulation Information
• (Problem Domain)
• Model
• FEDEP Products
• Federate VV histories Supplemental VV
  activities
• Federate Development Documentations
• System (Problem Domain)
• Develop Referent
• Expert Opinion
• Data
• Test Range
• Information Sources
• Related Systems
• Other Models
• Additional Info

10
Overview of Content

• 3. Plan VV
• (Under consideration of model system
  knowledge )
• Identify Priorities
• Identify Techniques Employed
• Put Together Team
• Identify Constraints to VV
• Admin (allocate resources, scheduling, etc)
• Guidance on how to plan VV activities

11
Overview of Content

• 4. Conduct VV
• (Provide evidence )
• Choosing Tools
• VV Report Development
• Dialog with testing community / Quality Assurance
  to ensure Technical Interoperability is provided
  (later phases)
• Support definition of test
• Substantive Interoperability / Compatibility
  (pragmatics, assess, semantics) of Federate
  Conceptual Model in integration of Federation
  Conceptual Model
• Conceptual Model reflects Referent and is
  suitable to achieve federation objectives. This
  is also true for scenarios.
• VV history (used in choosing federates) becomes
  relevant when you assess that the executable
  federate accurately reflects federate conceptual
  model. Map back history acquiring model
  information.
• Verify (and validate as possible) intermediate
  products against predecessors.
• Executable Federation tested for substantive
  interoperability problems.
• Create test scenarios complement testers
  scenario tests (create guidelines test for
  consistency)
• Example of common areas of inconsistency
• Listen to RTI traffic to observe Federation state
  changes
• Data (execute in conceptual model, in design,
  in executable)
• Results
• Data comparison with statistical techniques
• Visualisation (subject matter experts)
  Animation
• Automated data analysis

12
Overview of Content

• 5. Review Evidence
• Assessing sufficiency, credibility, probative
  force
• 6. Review Conducted VV
• like Accreditation Assessment (see
  RPG-Document from USA)
• Preparation for acceptability decision
• Sufficiency of assembled VV evidence
• 7. Residual Uncertainty Determination
• Risk

       
13
Issues

• Relationships of Processes
• Relationships between roles
• When does accreditation take place (not all
  nations perform)?
• Issues of levels and risk
• FEDEP not talking to substitute interoperability

14
Summary

• Continue dialogue
• Take into account work of other groups
• Eliminate duplication of effort