Improving Vendor Feedback in the RTI Verification Process

1
Improving Vendor Feedback in the RTI Verification
Process

• John Schloman, Jonathan Labin, Shon Vick
• john.schloman, jonathan.labin,
  shon.vick_at_jhuapl.edu

2
The RTI Verification Process
DMSO Verification
RTI Vendors
3
Current RTI Verification Process

• After a RTI verification run, deliverables
  returned to vendor are
• Three Adobe PDF documents per Phase of
  verification (5 Phases)
• For each of the failed tests
• Name of test
• Description of the test
• Verifier Analysis describing the anomaly
• Detailed log of verifier test output

4
Motivation for enhancement

• Error Log learning curve for new Vendors
• Unique format to verification software
• Adobe PDF
• Geared to human reading
• Small community of tools to read and manipulate
  documents
• Structure not conducive to data extraction,
  comparison, and archival.
• Reports are produced from MS Access 97
• No longer supported by MS
• Difficult to acquire maintain on latest
  Microsoft Operating Systems

5
An Augmentation

• Combined format
• Descriptive text from current PDF report
• Widely accepted structured data format
• A RTI verification extension to XML
• Popular and well-defined standard
• Wide community base of tools
• Designed to be both human and machine readable
• Construction tools distributed with Java
  (language of the verifier)

6
Specifics of Verifier XML

• Designed an XML schema
• Verification Error Markup Language (VEML)
• Based around HLA events that occur in
  verification tests
• calls, callbacks, exceptions, etc.
• Included all information currently distributed in
  report
• Verifier analysis, test description, etc.
• Additional meta-data
• Relevant information for tests and events are
  nested below the respective XML elements.
• Generation of VEML file in verification software
• Captures events during test execution
• Stores XML elements in Database for later VEML
  construction

7
Sample VEML file snippet
8
How does XML help?

• Existing XML tools facilitate generation of
  programs that use VEML file
• Classic vendor report generation
• Java tool translates VEML to LaTeX using XSLT
• Many tool choices for realizing LaTeX file to PDF
• Platform independence

9
Classic Vendor Report
10
VEML tools continued

• Vendor Replay Tool
• Java tool displays VEML content
• Failed tests displayed for selecting
• Text windows of analysis, test description etc.
• Replay of events
• User controlled step by step display of events in
  order
• Automated advancement of events (play)
• As events occur they appear on timeline and log
  window
• Visual and chronological mode of communication

11
Replay Tool
12
Potential VEML utilities

• Result Comparison Tool
• Locate test errors persisted across builds
• Locate tests that have changed error reason
• Highlight tests new to latest result set
• Could be built into a Vendors result archive
  utility to mine for trends or recurring problems
• Live Playback Replay Tool
• Similar to current Replay Tool prototype in
  display
• Instead of following VEML script of events,
  Federates are spawned to conduct the test against
  a live RTI
• Vendor can use Debug build of RTI to trace down
  errors quicker
• Vendor knows problem is solved before submitting
  next build

13
Questions? Suggestions?