Simone Youngblood

1
Decomposing the VVA Processes to Support Their
Tailoring

• Simone Youngblood S.Y. Harmon
• April 2008

2
Introduction
tailoring - the selection modification of VVA
activities to meet the needs of the intended use
within the risk, resource implementation
constraints of the program the simulation users

• This paper presents some intermediate results
  from an effort to develop guidance for risk-based
  tailoring of VVA processes.
• This effort assumes that any tailoring activity
  must begin with the cloth from which to select
  the needed VVA activities that will then be
  modified assembled into the process to be used.
• This paper describes a composite VVA process
  that represents this cloth from which practical
  VVA processes can be cut, modified assembled
  to meet use risk program resource constraints.
• Future work will supply detailed guidance to
  apply to the composite VVA process model (i.e.,
  the cloth) to produce tailored VVA processes.

3
A General Approach to VVA Tailoring

• An approach to tailoring verification, validation
  accreditation (VVA) processes
• Begin with a composite model that defines the
  entire spectrum of VVA activities tasks that
  could be done
• Select modify the components of that model to
  best satisfy the risk resource constraints of
  the simulation program
• Assemble the selected components into a
  consistent process that satisfies the use risk
  goals of the simulation program while still
  adhering to the policies, standards or guidelines
  that may be relevant to the intended use
• Practitioners will tailor this composite VVA
  process model to meet the needs constraints of
  their particular situations.

Select Modify the VVA Activities Tasks to
Meet Project Resource Constraints
Assemble the Selected Activities Tasks into a
Process that Meets the Projects Use Risk Goals
Execute the Tailored VVA Process Modify It as
Project Constraints Goals Change
4
Assumptions Underlying the Composite VVA Model
Four assumptions underlie the construction of the
composite VVA process model

• This general model represents the breadth of the
  VVA processes from user requirements to the
  acceptance decision for an intended use.
• Accreditation refers to the process for arriving
  at a decision to accept or accredit a simulation
  for an intended use.
• The primary purpose for performing verification
  validation (VV) is to collect the evidence
  needed to support an acceptance decision for an
  intended use.
• The simulation users or their representatives are
  responsible for ensuring the completeness
  correctness of any statements of user needs or
  simulation requirements.

5
Contributing VVA Process Models
The composite VVA process model was constructed
by identifying the activities tasks in many
other VVA process models aggregating a
selected union of those components.
Simulation VVA Process Models
Federation VVA Process Models

• Sargent VV Process (2007)
• Balci VVA Process (1998)
• U.S. DoD Problem Solving Process (2006)
• Navy VVA Process (2004)
• U.K. VV Process (2001)
• Canadian VVA Process (2003)
• Australian VVA Process (2005)
• Brade Generalized VVA Process (2000)
• REVVA 1 VVA Process (2004)
• Stroud VVA Process (2005)
• Goerger HBR VVA Process (2003)

• DIS VVA Process (1995)
• Graffagnini et al. VVA Overlay (1999)
• Topcu Federation VV Process (2003)
• Dobey Lewis VVA Overlay (2003)
• IEEE VVA Overlay (2007)

Software VV Process Models

• IEEE Software VV Process (2004)
• German V-Model (2008)

The following slides present a sample of those
models considered in constructing the composite
VVA model
6
Example Early Model of VV Processes Proposed by
Sargent (Sargent 2007)
R.G. Sargent, Verification and Validation of
Simulation Models, Proc. 2007 Winter Simulation
Conf., Washington, DC, 9-12 December 2007,
p124-137.
7
Example Precursor to DOD VVA Process
Model(Balci 1998)
O. Balci, Verification, Validation, and
Accreditation, Proc. 1998 Winter Simulation
Conf., Washington, DC, 13-16 December 1998,
pp41-48.
8
Example DoD MS Problem Solving Process
Model(US DoD 2006)
Defense Modeling and Simulation Office,
Verification, Validation and Accreditation (VVA)
Recommended Practices Guide (RPG), Build 3.0,
Department of Defense, Alexandria, VA, September
2006.
9
Example Navy VVA Process Model(NMSO 2004)
Navy Modeling and Simulation Office, Modeling and
Simulation Verification, Validation and
Accreditation Implementation Handbook, Volume I,
VVA Framework, Department of the Navy,
Washington, DC, 30 March 2004.
10
Example UK VV Process Activities(UK Defence
Std 03-44)
dstl, UK Defence Standard 03-44 on the
Verification and Validation of MSSE, Ministry of
Defence, London, UK, 2001.
11
Example Canadian VVA Process Model(SECO 2003)
Synthetic Environment Coordination Office,
Modelling and Simulation Verification, Validation
and Accreditation (VVA) Guidebook, Ver. 0.0,
Canadian Department of National Defence, Ottawa,
Canada, May 2003.
12
Example Australian VVA Process Model(ADSO 2005)
Australian Defence Simulation Office, Simulation
Verification, Validation and Accreditation Guide,
Department of Defence, Canberra, Australia, 2005.
13
Example REVVA 1 VVA Process Model(Brade 2004)
Create ToA and determine acceptable residual
uncertainty
Get overview over available model information and
available real system knowledge
Plan VV and Derive ToVV (Decomposition)
Assess VV (Re-composition)
Assess Evidence
Implement VV
D. Brade, WP1 in JP 11.20, Results Overview,
Swedish Defence Research Agency, March 2004.
14
Example Nine Step VVA Process Model for
DIS(Sandmeyer 1995)
R.S. Sandmeyer et al., Comparison of VVA for
A2ATD Experiment One to DMSO Nine-Step Process,
Paper No. 13-95-039, Proc. 13th Distributed
Interactive Simulation Workshop, Orlando, FL,
18-22 September1995, np.
15
Example IEEE VVA Overlay Process Components
(IEEE Std 1516.4-2007)
IEEE Computer Society, IEEE Recommended Practice
for Verification, Validation, and Accreditation
of a Federation - An Overlay to the High Level
Architecture Federation Development and Execution
Process, IEEE Std 1516.4-2007, IEEE, New York,
NY, 20 December 2007.
16
Proposed Composite VVA Process Model
The accreditation process encompasses the VV
process needed to supply the validation evidence
from which to derive the acceptance
recommendations.
17
Proposed Composite Accreditation Activities
Tasks

• Plan the Accreditation Effort
• Assess the Accreditation Needs
• Develop the Accreditation Approach
• Build the Accreditation Schedule
• Estimate the Accreditation Costs
• Prepare the Accreditation Plan
• Execute Evolve the Accreditation Plan
• Characterize the Simulations Use
• Collect the User Needs
• Verify the Requirements/Objectives
• Document the Intended Use
• Assess the Use Risks
• Define the Conditions of Expected Use
• Develop the Acceptability Criteria
• Specify the Accreditation Information
  Requirements
• Construct the Validation Referent
• Identify Practical Referent Information Sources
• Collect the Applicable Referent Information
• Assemble the Validation Referent

• Collect the Validation Evidence
• Plan the VV Effort
• Apply Relevant Historical Information
• Verify Validate the Simulation Conceptual Model
• Perform Supplementary Verification
• Leverage the Verification Products
• Verify Validate the Data Knowledge Sets
• Validate the Simulation Results
• Integrate the Validation Evidence
• Develop the Acceptance Recommendations
• Analyze the Validation Evidence
• Develop the Acceptance Recommendations
• Prepare the Accreditation Report Accreditation
  Support Package
• Support the Official Decision to Accredit

18
Composite Process for Collecting Validation
Evidence
19
Process Components for Collecting Validation
Evidence

• Plan the VV Effort
• Develop the VV Approach
• Build the VV Schedule
• Estimate the VV Costs
• Prepare the VV Plan
• Execute Evolve the VV Plan
• Apply Relevant Historical Information
• Collect Analyze the VV Testing History
• Collect Analyze Prior Use History
• Collect Analyze Developer Accounts of
  Simulation Capabilities Limitations
• Identify Factors that May Constrain Use from the
  Historical Information
• Integrate the Historical Evidence
• Identify Effective Output Sampling Areas from the
  Historical Evidence
• Verify Validate the Simulation Conceptual Model
• Characterize Conceptual Model Coverage
• Check the Internal Consistency of the Conceptual
  Model
• Infer the Intended Representational Capabilities
  from the Conceptual Model
• Evaluate Conceptual Model Validity
• Verify Validate Available Scenarios

• Leverage the Verification Products
• Characterize Collective Verification Coverage
• Infer the Representational Capabilities from the
  Verification Products
• Evaluate Development Product Validity from the
  Verification Products
• Identify Effective Output Sampling Areas from the
  Verification Products
• Identify Factors that May Constrain Use from the
  Verification Products
• Integrate the Verification Evidence
• Verify Validate the Data Knowledge Sets
• Identify the Data Knowledge Sources Their
  Pedigrees
• Find Authoritative Sources for Data Knowledge
  with None
• Verify the Internal Consistency of Data
  Knowledge
• Verify Data Transformations
• Validate Data Knowledge Sets Where Needed
• Identify Effective Output Sampling Areas from the
  Data Knowledge Sets
• Identify Factors that May Constrain Use from Data
  Knowledge VV
• Integrate the Data Knowledge VV Evidence
• Validate Simulation Results
• Plan for Results Validation
• Leverage Developer Test Results for Validation

20
Steps toward Building Risk-Based VVA Tailoring
Guidance

• Develop the VVA process cloth
• Build a composite VVA process model
• Vet this process model in the VVA community
  refine it from the feedback received
• Develop detailed description of the tasks in the
  process model
• Identify the relationships that must exist
  between the VVA process activities tasks
• Vet the VVA task descriptions relationships in
  the VVA community refine them from the
  feedback received
• Develop descriptions of the VVA product quality
• Vet the descriptions of VVA product quality in
  the VVA community refine them from the
  feedback received
• Develop the VVA guidance
• Identify the tailoring decisions about task
  selection, assembly modification
• Identify the factors upon which the tailoring
  decision depend
• Vet the tailoring decisions factors in the VVA
  community refine them from the feedback
  received
• Develop strategies for tailoring the VVA process
  based upon risk resource constraints
• Vet the VVA tailoring strategies in the VVA
  community refine them from the feedback
  received
• Assemble the process model, tailoring decision,
  factors strategies into coherent consistent
  VVA tailoring guidance
• Vet the VVA tailoring guidance in the VVA
  community refine it from the feedback feedback
  received

21
Draft NMSG-054 Timeline for Developing Guidance
for Risk-Based VVA
refined VV task descriptions initial VV task
relationships
refined tailoring factors initial tailoring
strategies
final tailoring guidance
refined VV task quality description initial
tailoring factors
refined VV process model VV task descriptions
delivered NMSG final report
refined draft tailoring guidance
refined VV task relationships VV task quality
description
refined tailoring strategies draft tailoring
guidance
composite VV process model
draft NMSG final report
CY 09
CY 08
22
Summary Conclusions

• This comprehensive model of the VVA processes
  describes ALL of the activities tasks that
  COULD be performed to meet a simulation programs
  accreditation needs.
• This model is not intended to be applied in
  totality or universally but should be tailored to
  meet the resource use risk constraints of
  particular situations.
• This model only supplies the cloth from which to
  cut the VVA processes needed to suit the demands
  resource availability of specific simulation
  programs.
• VVA practitioners are expected to use their
  knowledge of the risks that their simulation
  users can cannot accept to decide which VVA
  activities tasks to perform at which the
  maturity level to perform them.