A Perspective on Formal Certification

1
A Perspective on Formal Certification
Dr. Lee Whitt Technical Director Defense Mission
Systems Navy Group October 20, 2004
2
Discussion Topics

• A Topical Case
• Why Formal Certification?
• OASDs Checklist for NCW
• NCOC role

Todays Application is Tomorrows Infrastructure
3
A Topical Case Transformational Path for Navy
C4ISR
Taking a page from Darwins Theory of
Evolution Its not the strongest or fastest
that survive, but rather the most adaptable to
conditions.
You are here
NTCS-A OSS
Stove pipes
JC2
GCCS-M 4.x
GCCS-M 3.x
JMCIS
JOTS-II
JOTS-I
20 years
4
A Topical Case GCCS-M 4.x

• Requirements
• Adaptable for diverse military operations
  supporting COIs on a global scale (from planning
  through execution through replay/reconstruction
  post-analysis)
• Network-centric distributed computing
  environment, leveraging open standards and open
  architecture
• Fault-tolerant and sustainable in austere
  conditions (e.g., platform-centric when
  necessary)
• High performance, secure, and bandwidth efficient
• Easy to use
• Backward compatible with deployed C4ISR systems

5
A Topical Case GCCS-M 4.x

• Target environment
• All US Navy ships (big deck small deck),
  submarines, shore command centers, and a few
  aircraft including many coalition/allied Navies
• Multiple security enclaves (e.g., Secret, Top
  Secret, Coalition-releasable)
• Target configurations
• Large LANs (100 w/s 5 servers), small LANs
  (10 w/s 2 servers), and mobile LANs (for
  special operations)
• Multiple w/s configurations (6 specialized
  according to mission) multiple server
  configurations (3 specialized)
• Multiple communication/interface configurations,
  depending on platform
• Over 300 packaged application segments
• Heterogeneous LANs consisting of Solaris and
  Wintel, with occasional Linux including
  different hardware units and OS versions
• Target interfaces
• All major Navy sensors, combat systems, message
  formats, and protocols
• Legacy C4ISR systems (e.g., GCCS-M 3.x)
• Target User 19 yr old sailor, 4-Star Admiral,
  and everyone in between

6
What is the Test Plan?

• Segment Test Plan
• Responsibility of Segment Developer Segment
  Program Office
• Integration Test Plan
• Responsibility of System Integrator GCCS-M
  Program Office
• Operational Test Plan
• Responsibility of OPTEVFOR (Operation Test
  Evaluation Force)

7
What is Missing?

• Test methodology doesnt scale
• C4ISR system functionality and complexity has
  mushroomed, but test process hasnt kept pace
• Use cases are a band-aid approach to end-to-end
  testing
• Test methodology doesnt address adaptability
• Composeability presents unique challenges and new
  levels of complexity use cases are not
  sufficient
• No participation of component developers in
  system-level test
• System success is responsibility of everyone

8
Why Formal Certification?

• Establishes repeatable defendable criteria for
  system evaluation
• Focuses on expected behavior, edge cases, and
  anomalous conditions
• Focuses on quantifying interoperability
• Promotes periodic evaluation of test process
• Ensures appropriateness of test criteria for the
  target system
• Focuses attention on modeling system behavior
• Suggests Levels of Certification/Compliance
• Provides achievable goals
• Promotes migration strategies to higher levels
• Fosters instrumentation of system components
• Designers need to formulate component models
  Developers need to define expose monitor points
• Facilitates regression testing

9
What should be tested?

• Different system views variations on the
  theme
• TV
• OV
• SV
• Different stages of software life-cycle
• Compile time, Deploy time, Run time,
  Trouble-shooting time
• Dynamic diverse forms of adaptability/composeabi
  lity
• What test methodology can quantify this
  attribute?
• Open standards, open architecture,
  interoperability, etc.
• How do we quantify these concepts and craft test
  regimes?

10
Compliance Checklist from OASD

• Checklist for PMs to
• Identify network-centric attributes of programs
  for participation on the GIG
• Define criteria for compliance with
  network-centric principles

11
Network-Centric Checklist

• Four Sections
• Data
• Services
• IA/Security
• Transport

12
Network-Centric Checklist (selective)

• Data
• Is the data tagged for discovery?
• Are web services used to make the data available?
• Are alerts available for new or modified data?
• Can the data be sanitized to cross security
  boundaries?
• Is the data access controlled?
• More.
• Services
• Are web services based on industry standards (3
  page list of standards)?
• How scalable and fault-tolerant are the services?
• How is bandwidth managed?
• Is the service instrumented to provide status and
  audit trails?
• More.

13
Network-Centric Checklist (selective)

• IA/Security
• Is identity management and authentication used?
• Are security assertions mediated?
• Is High Assurance IP Encryption (HAIPE)
  supported?
• Is auditing available to conduct security checks?
• Is secure IA management provided to protect the
  GIG?
• More.
• Transport
• Are both IPv4 and v6 supported, along with
  transition technologies (e.g., tunneling,
  dual-stack)?
• Is QoS provided?
• More.

14
Where to from Here?
15
NCO Consortium

• NCOC offers unique opportunity to address
  many certification issues
• Open Group leadership is key
• Coordination with DoD certification facilities is
  required
• Industry and govt dedicated support is required
• Phased approach needs to be defined
• Scoped for success
• Focused on next-generation systems
• Establish a distributed vendor
  neutral network of IVV labs
• Provide a level playing field for DoD industry
• Include contractual language to force compliance

16
Final Comments

• Formal certification is critically important for
  C4ISR systems, but it must span the entire
  system stack
• End-to-end certification is the real requirement
• System complexity is on the rise and new
  testing/certification concepts are needed
• Todays applications are tomorrows
  infrastructure
• The Open Group and NCOC offers a vendor-neutral
  approach

17
Backup Slides
18
Another Topical Example E-Voting Machines

• Testing certification problems abound
• Lack of technical standards for e-voting machines
• Lack of transparency in testing certification
• What are the credentials of the testers?
• What is the test plan?
• What are the test results?
• Lack of completeness in testing certification
• and this is an easy application

Broward County 2004 E-voting machines lost 134
votes margin of victory was 12 votes.