Betsy B' DeLong

1

DEPARTMENT OF THE NAVY MODELING AND SIMULATION
MANAGEMENT OFFICE
Verification and Validation Ensuring Data
Credibility
2001 European Simulation Interoperability
Workshop University of Westminster, Harrow
Campus Harrow, Middlesex, UK June 27, 2001

• Betsy B. DeLong
• VVA Program Manager
• Delong.Betsy_at_hq.navy.mil
• [phone]

2
Why use credible data?

• Design and use of any technology (especially
  MS) requires the proper collection and use of
  data.
• Molasses Tank Disaster in Boston harbor
  (United States of America), January 1919

3
Data Collection

• Models composed of physical, behavioral, and
  environmental entities.
• Data collected from -
• manufacturers, developmental and operational
  testing, operational use, or special
  organizations (physical)
• organizational doctrine, recording behavior,
  conducting extensive interviews with
  knowledgeable individuals (behavior)
• variety of sources under a variety of
  environmental conditions (environmental)

4
Data Use and Credibility

• Many types of data go into a model or simulation.
• Watch out for
• coordinate reference transformations
• constraints imposed by designer
• This can be especially difficult in distributed
  simulations or federations!
• Data credibility is dependent on
• how it is produced and maintained
• how it is transformed and used within a MS
  application.

5
DoD and Navy Policy

• Department of Defense (DoD) Directive 5000.59
• mandates verification and validation (VV) of
  data used for modeling and simulation (MS).
• Navy MS Master Plan
• authorized by Secretary of the Navy Instruction
  (SECNAVINST) 5200.38
• main objective is development of a synthetic
  battlespace populated with credible MS and data
  of the highest quality.
• DoD Instruction 5000.61
• VVA Recommended Practices Guide, Millennium
  Edition
• SECNAVINST 5200.40
• Department of the Navy (DON) VVA Implementation
  Handbook

6
Department Of the Navys Steps to Data Validity

• DON VVA Program and Process
• Taxonomy for authoritative MS data sources
• DoD MS Executive Agent (MSEA) for Oceans and
  Littoral Warfare Environments
• Data Standards Technical Area Group

7
Defining MS Data
MS Authoritative Data Sources (ADS) Program
provides a centralized repository of data and
data sources for MS users and developers.
8
MS Data VV
DON Data VV Activities
DON VVA PROCESS
1. Verify required data
Preliminary Activity - specify analysis
requirements
2. Verify data source availability. 3. Verify
metadata accuracy. 4. Verify input database. 5.
Verify output characteristics. 6. Develop
validation data.
Conceptual Model Validation
7. Plan data VV activities.
VV Plan
8. Validate data transformation methods 9.
Verify transformed instance data 10. Verify
hardwired data
Functional Verification
System Verification
11.Verify initialized data 12. Validate hardwired
data 13. Verify output
Results Validation
14. Validate data 15. Adjudicate errors
9
DON Data VV Activities

• 1. Verify Required Data
• What are the requirements for the data?
• How will the data be used in the MS?
• 2. Verify Data Source Availability
• Can data be obtained from an authoritative
  source?
• Does the required data exist? Is it available for
  use?
• 3. Verify Metadata
• Is the data current?
• Does it have the proper level of fidelity?
• Are the correct units of measure applied?

10
DON Data VV Activities

• 4 5. Verify Input Databases and Output
  Characteristics
• Is the data complete?
• Can I trace the data to the user requirements
• 6. Develop Validation Data
• Are the results consistent with comparable
  real-world scenarios?
• Can subject matter experts differentiate between
  the MS results and the real-world outcome being
  simulated?
• 7. Plan Data VV activities
• Data VV Requirements Data VV Methods
  Requirements Traceability

11
DON Data VV Activities

• 8 9. Validate Data Transformation Methods and
  Verify Transformed Instance Data
• Can the data be used as is or will they have to
  undergo transformations (i.e., feet to meters,
  etc.)?
• 10. Verify Hardwired Data
• Have the hardwired data been fully documented?
• Is the implementation consistent with the
  documentation?
• 11. Verify Initialized Data
• Are the data transformations documented?
• Are the original data traceable to the
  initialized, or instance, data?

12
DON Data VV Activities

• 12 13. Validate Hardwired Data and Verify
  Output Data
• Is the algorithm providing accurate and
  appropriate outputs for the given inputs?
• 14. Validate Output Data
• Are all the necessary data transformations well
  documented?
• Have the transformations been traced to where
  they are implemented?
• 15. Adjudicate Errors
• Have all possible errors been accounted?
• Have uncorrectable errors been documented?

13
EXCHANGE DATA VV
Exchange data is data that defines how
information is passed between federates.
14
MS Data Accreditation

• Determine data requirements
• What are the requirements for the data?
• How will the data be used in the MS?
• Determine acceptability criteria
• What criteria must be met to accredit the data
  for use in the MS application?
• What level of risk can the Accreditation
  Authority live with?
• Make Accreditation Decision
• Document limitations
• Primary assurance of MS data quality is
  accreditation of the MS and its data.

15
SUMMARY

• In the Defense arena, it is vitally important
  that the data used be the right data.
• Lives depend upon it!

16
CONTACT INFORMATION

• Betsy B. DeLong
• VVA Program Manager
• DONMS-TSG (OPNAV N6MT2)
• 2511 S. Jefferson Davis Highway
• Arlington, Virginia, USA, 22202
• Phone [phone]
• Fax [phone]
• E-Mail DeLong.Betsy_at_hq.navy.mil
• E-Mail vva_at_navmsmo.hq.navy.mil