What

1
Whats the Exit Strategy?

• Verifying a Design

2
Agenda Design Verification

• Concepts and implications
• The specification connection
• Verification before test
• Test thoroughness
• Tiered verification
• Concluding the process
• Summary

3
Verification Concepts

• Verification Confirmation, through the
  provision of objective evidence, that the
  specified requirements have been fulfilled
  (Q9000-2000)
• Confirmation the act of ratifying or
  corroborating (AH)

4
Verification Implications

• The verification process is not intended to be a
  craps shoot
• Verification is primarily intended to ratify
  correctness
• Verification is NOT primarily intended to reveal
  incorrectness
• The mindset is important!
• Doubt that a design will work expresses a lack of
  confidence in the designer and the design process
• Waiting until verification to guarantee
  correctness is an excuse for being sloppy
• We should expect our designs to work!
  Verification simply puts the seal of confirmation
  on the expectation

5
Verification Implications (cont.)

• Verification addresses two processes
• Design
• Primarily a one-time, iterative process in which
  the developed concept is proven sound
• Fundamental correctness can be proven
  analytically
• Inspection confirms logical correctness in all
  circumstances
• Peer reviews are a manual form of inspection
• Functional simulation is an automated form of
  inspection
• Inspections must be tailored to design
• Analysis verifies correctness in the presence of
  variability
• Reliability (WC, Derating, FMEA, etc.)
• Timing over environments and age

6
Verification Implications (cont.)

• Verification addresses two processes (cont.)
• Instantiation
• Particular instance must be sound
• Particular correctness must be demonstrated
  experimentally
• Inspection confirms that instructions are
  followed
• Correct components installed
• Correct configuration selected
• Correct processes performed
• Test and demonstration confirms that operation
  matches expectations
• Functionally and parametrically
• Predicted by nominal analytical predictions

7
Verification Implications (cont.)

• Verification after the fact is too late
• Analysis and test are NOT equivalent
• Design analysis and inspection must proceed in
  lockstep with design processes
• Implementation tests and inspections should
  simply confirm what is known
• Design efforts fail because they occur in a
  vacuum
• Creativity takes primacy over correctness
• Functionality comes first with operability being
  shoehorned after the fact
• The monster simulation syndrome
• Conclusion Verify as you proceed

8
Verification Implications (cont.)

• Correctness is a matter of personal integrity for
  the designer
• Verification is not simply externally imposed
  task
• Verification is something that the designer must
  want to do (a part of his/her ethos)
• Verification is confirmation that the designer is
  worth his/her salt

9
The Specification Connection

• Requirements are confirmed during verification
• The designer does not have a free hand with
  respect to how a design is confirmed
• Specification provides the constraint set under
  which verification is prosecuted
• Therefore, verification must be defined prior to
  start of design
• Not simply in a general manner
• Specifically
• Since specification is a customer and vendor
  document
• Both customer and vendor must be involved in the
  verification process
• Trust me is not a reasonable phrase when it
  comes to verification

10
The Specification Connection (cont.)

• Specification contains a complete ideally set
  of requirements for the design
• Some requirements are very specific
• All discrete outputs shall have a source
  impedance of 2 kOhms.
• An adequate test is fairly obvious
• Some requirements are not very specific
• The unit shall provide 512 kbytes of SRAM
• Note the implied requirement The SRAM shall
  function correctly
• What is an adequate functional test for this
  requirement? (walking 1s, 0s addressing, etc?)
• Some requirements lend themselves to quantitative
  analysis The unit shall provide a .1 C
  accuracy at end of life
• Some requirements lend themselves to simple
  inspection The unit shall provide 4 analog
  output channels
• When do we decide adequacy of the method of
  verification and the individual verification
  cases? Before we start designing

11
The Specification Connection (cont.)

• Therefore, verification planning must begin with
  specification creation
• Each requirement must be assigned a method or
  methods
• Each requirement must be assigned a test or
  analysis case
• Must answer question What is an adequate
  verification?
• Must establish answer to question When are we
  done?
• Each requirement must be verified at one or more
  levels FPGA, board, box, sub-system, system
• Customer must concur with decision
• Note that modern verification databases make this
  relatively straightforward

12
The Specification Connection (cont.)

• Advantages to this type of verification planning
• More verifiable / verified designs
• Up-front focus on programmatically difficult
  verification can be instituted
• Earlier analysis
• Simplification of overly complex circuit designs
• Guidance for lower-level verification efforts can
  be established
• Sub-module vs. module level
• Module vs. unit level
• Integral self-test provisions can be included in
  design
• Earlier simulation vector definition (FPGAs /
  logic)

13
The Specification Connection (cont.)

• Advantages (cont.)
• More robust test sets
• Early knowledge regarding end-item function
  communicated
• Capability of test set matched to need
• Precision (timing, voltage, current, etc.)
• Speed
• Test level (component, unit, system)
• Test flow design considerations included in test
  set design
• Knowing verification requirements early makes the
  entire development process more efficient

14
The Specification Connection (cont.)

• Defining requirements and test cases at the same
  time as the specification clarifies and
  simplifies the FPGA verification process
• Allows early start to simulation vector design
• Creates early knowledge of additional functional
  models (SRAM, peripherals, etc.) needed
• Clarifies decision regarding what can be
  functionally simulated and what must be simulated
  with back-annotation
• Defines levels at which simulations must be run

15
Verification Before Test

• It is a relatively bad thing to discover a design
  flaw during test (better than nothing)
• Late in the development cycle
• Correction is more complicated / expensive
• Personal stress is higher
• Yet a surprising lack of verification occurs
  early
• Types of pre-test verification
• Inspection - conformity evaluation by observation
  and judgment accompanied, as appropriate by
  measurement or testing (ISO)
• Personal self-check
• Peer Review
• Basic functional verification (Signal audits,
  functional simulation, etc.)

16
Verification Before Test (cont.)

• Types of pre-test verification (cont.)
• Analysis Conformity evaluation of the
  predictions produced by realistic models of the
  system
• Worst case analysis (voltage, current, frequency,
  time) using models that incorporate real world
  degradation of function
• Derating analysis using models that reduce stress
• Other
• Note that pre-test verification is fundamentally
  conceptual
• Not tool dependent
• Not design dependent
• Can be accomplished many ways

17
Verification Before Test (cont.)

• Two approaches to pre-test verification
• Verify full design at once (one big peer review,
  code walkthrough, analysis, simulation)
• Benefits
• Complete design reduces need to develop
  assumptions
• When proved correct, does not need to be repeated
• When design is solid, less work
• Reduce final documentation for verification
• Drawbacks
• Allows small design flaws to propagate for a long
  time
• Complexity reduces visibility (verification more
  prone to mistakes and oversights)
• When design is flawed, it increases work
• Leads to corrections that are global (hard to
  test effectively and predict side effects)
• May take longer to execute than an aggregate of
  smaller verification activities

18
Verification Before Test (cont.)

• Two approaches to pre-test verification (cont.)
• Verify incremental designs
• Benefits
• Supports development of known good sub-designs
• Meshes with a partitioned design approach
• Facilitates visibility (fewer mistakes, clearer
  goals)
• Allows small flaws to be fixed quickly (minimizes
  downstream impact)
• Drawbacks
• Is more work in the ideal case (no/few flaws)
• Increases documentation if formality is vital
• Either approach can be made to work, but one or
  the other must be chosen and implemented

19
Test Thoroughness

• Principles for test thoroughness
• Every shall in the specification that meets the
  following criteria must be tested
• Items that may be affected by the instantiation
  process (subject to fabrication error)
• Items for which the only extant verification is
  model based
• Items for which the testing does not pose
  unacceptable risk (radiation, overstress, etc.)
• Every instance of a shall must be tested
• 18 interfaces require 18 specific tests
• Requirements that have an exclusive character
• Must be tested for correct operation
• Should be tested at some level for abnormal
  operation
• Example If you write a 5Ah to something to set
  it on, you should verify that writing other
  values does not set it on
• Requirements must be tested over a representative
  range of conditions

20
Test Thoroughness (cont.)

• A thorough test is complicated and time consuming
• Requires some level of automation to be
  comprehensive
• Requires some level of compromise to be practical
• Requires team agreement to be acceptable

21
Test Thoroughness (cont.)

• Ensuring thorough, yet practical, tests
• Define which tests require manual interaction and
  which can be automated
• Output impedance or analog fine precision may
  need to be manual
• Signal level or analog coarse precision may be
  automatable
• Define which tests must be performed over
  environmental conditions and which can be
  performed at room temperature only
• Make decision based on character of measurement
• Do not decide based purely on practicality
• Define level of complexity for all measurement
  sets
• Example 24 analog inputs (3 eight channel muxes)
• Test full range of values for 24, or limited
  range for 7 of 8 mux inputs and full range for
  remaining
• Define need for repetition of measurement at
  higher level of integration (tiered verification)
• Example voltage level / signal quality / clock
  jitter at board level
• Repeat at box level?

22
Test Thoroughness (cont.)

• Test thoroughness issues affect everything
• Customer relationship
• Test set design (board, box, system)
• Schedule / cost
• An adequate test program is not trivial
• Cannot wait until system level
• Must incorporate lowest level of design
• Must be a constant background activity during
  design process
• All programs must balance perfect and good enough
• A good test program
• Will ensure that the specification is met
• Will verify everything that can be affected by
  fabrication
• Will build on the analysis and inspection effort
• Will maximize automation without sacrificing test
  accuracy

23
Tiered Verification

• Tiered verification is the process of matching
  the correct type of verification to the
  appropriate level of integration
• Tiered verification incorporates all types of
  verification (before and during testing)
• Tiered verification applied to the test regime
  attempts to have thoroughness with practicality
• Test a requirement at the lowest conceivable
  level
• Determine which tests must be repeated at higher
  levels by
• Establishing the purpose of a test at a
  particular level
• Determining whether the next level has the
  potential to change previous measured quantities
• Determining what test set capabilities are
• Tiered verification cannot be retrofitted into
  the process
• Must be planned up front
• Must be implemented throughout the development
  process

24
Tiered Verification - Example

• Need A set of 16 high-level pulsed discretes to
  control latching relays used to change the
  spacecraft power configuration
• Basic Requirements
• Quantity 16
• Level 30V /- 2 V
• Load 2 kOhm, 4 mH
• Duration of pulse 50 ms
• Rise / Fall time max 1 ms
• Command capability Unique bit pattern, only one
  at a time, arm first
• S/W requirements Various (beyond scope of
  example)

25
Tiered Verification Example (cont.)

• Tier 1 Basic Verification (functional
  simulation)
• Verify each channel fires for a specific bit
  pattern / address
• Verify cannot be fired without an arm command
• Verify that other complementary patterns dont
  accidentally fire the pulse
• Analyze drive capability of FPGA and input /
  output requirement / capability of translation
  chip
• Evaluate glitch hardness of logic selected

26
Tiered Verification Example (cont.)

• Tier 2 Board level
• Test to confirm signal level between FPGA and
  driver chips is compatible
• Test rise, fall, duration, level with dummy load
• Repeat test for all 16 outputs
• Replace dummy load with relay, verify that the
  relay switches for all 16 outputs
• Repeat over temperature if necessary

27
Tiered Verification Example (cont.)

• Tier 3 Box level
• Attach GSE (verifies with relays)
• Functionally pulse all outputs
• Verify relay switches
• Verify gross drive duration
• Tier 4 Box and operational software level
• Verify functionality commanded through software
• GSE verifies correct relay switch
• Tier 5 System
• Verify gross functionality as needed for system
  operation

28
Tiered Verification - Notes

• Most complex and detailed functionality is
  verified at lower levels
• Performance
• Error cases
• Predictive of future operation
• Higher levels focus on functionality / operation
• Lower level verification requires more manual
  touch
• Higher level verification is more automated
• STE and test procedure developed as appropriate
• Purpose is maximum test completeness
• As well as bang for the buck

29
Concluding the Process

• ( through the provision of objective
  evidence)
• The importance of traceability
• Something is only verified (formally) when it is
  documented
• The tie between verification item and
  specification must be made (verification matrix
  or database)
• The importance of planning
• Establish the links between requirement, method,
  level, test case early
• Use the established structure to develop
  verification items
• The importance of buy in
• All responsible must complete verification and
  report results
• Systems engineers must track and close
• People must keep up with the process
• Only personal commitment from all involved will
  ensure that the process is successful

30
Summary

• Verification is to prove success, not avoid
  failure
• Verification planning is an integral part of the
  project lifecycle
• Must be initiated with specification (including
  customer buy-in)
• Must be included in design effort
• Must be used to develop documentation and
  appropriate test hardware
• Verification is more than final test
• Incorporates analysis and inspection throughout
  design process
• Begins at lowest possible level
• Develops proof of compliance at all levels of
  operation
• A thorough test is complicated
• Requires extensive work
• Must trade perfect for practical (a tiered test
  approach can help)
• Verification processes must be tracked and
  documented
• The whole team must buy in