CxSI Validation Results (Using a Combination of New and Traditional IV

1
CxSI Validation Results (Using a Combination of
New and Traditional IVV Analysis Techniques)

• Objectives
• Summarize Validation Results
• Convey processes and plans used by CxSI IVV
• Provide a context for discussion about modeling
  and validation

christina.d.moats_at_nasa.gov cxsi_team_at_ivv.nasa.gov
2
Validation Report Summary
Analysis Results Summary (57 TIMs total)
3
IVV Issues mapped to Model
Sortie issues impact Lunar Base mission -
requirements in draft
Mission comparison found missing Mars requirements
Hazard detection requirement not flowed down to
SRD
Missing alternate undock requirement
Unclear backup S-Band requirement impacts
all behaviors
Severity 3
Severity 2
Severity 4
IVV Activities
4
CxSI-AD01.02 R1.0 2007-07-27
Activity Diagram
Use Case
CxSI-UC01.02 R1.0 2007-07-27

• Travel to Moon - pg. 1 of 2

Orion
Crew
Mission Operator
LSAM
EDS
Open Docking Hatches
Perform on-board EDSLSAMOrion preparations for
TLI
Conduct EDSLSAM Orion preparations for TLI
Perform Mission Operator Procedures
Must work capability missing alternate undock
requirement
Loss of comm
Else
Else
no go
Return to Orion and secure hatches
undock from EDSLSAM
Conduct TLI maneuver for EDSLSAMOrion
Dispose of or perform contingency mission with
EDSLSAM
Land on Earth Ref AD04.05
anomaly correction NOT possible
Else
Else
time critical or not on free return
anomaly correction possible
Land on Earth Ref AD04.05
Use delta V required to return to Earth (w/ Orion)
Undock from LSAM
Exceeds ?V budget
Correct trajectory
Else
retain the LSAM as a lifeboat
Conduct lunar observations
Separate Orion LSAM from EDS
EDS separation Anomaly
Else
A
5
Retrospective on the CxSI Analysis

• Completing integration analysis sets a stage for
  lower level CxP system analyses
• Both processes and findings
• Observations
• Significant issues/challenges exist w.r.t. L2
  requirements
• The CARD L2 requirements are the foundation for
  the entire CxP and must be identified/defined/trac
  ed with greater rigor and accuracy

• A combination of the old and the new analysis
  techniques was useful
• Behavior Analysis (using SRM)
• Probably the most significant single finding
  (1067, assured undock capability during reentry)
  of our analyses was identified as part of this
  analysis
• Model based analysis identified 6 TIMs, 5 missing
  requirements (10 total findings) few issues
  were found, but they were tricky issues
• The remaining tasks resulted in 90 of the
  submitted TIMs

6
Current Efforts follow plans developed in March 07

• Three tasks will complete by July 27, 2007
• Develop Behavior Model (SRM),
• Analysis Preparation
• Validation Methods

Validation analysis will occur from July - Sept
2007
Note this process will be expanded in future
slides
7
CxSI Behavior Model Generation and L2 Validation
Process based on this Model
Feb-July, 2007
Aug-Sept, 2007
Note this process will be expanded in future
slides
8
How do we validate our model?

• The CxSI SRM
• Dependability Criteria
• SRM Readiness to support Requirements Validation
• Requirements Assessment
• Peer Review
• SRM Best Practices/Lessons Learned

9
Constellation Behavior Model Hierarchy
CxSI SRM addresses four primary goals ISS, Moon
Sortie, Moon Base, Mars
10
Dependability Criteria

• Dependability criteria
• Availability The probability that a system is
  operating correctly and is ready to perform its
  desired functions.
• Consistency The property that invariants will
  always hold true in the system. 
• Correctness A characteristic of a system that
  precisely exhibits predictable behavior at all
  times as defined by the system specifications. 
• Reliability The property that a system can
  operate continuously without experiencing a
  failure. 
• Robustness A characteristic of a system that is
  failure and fault tolerant. 
• Safety The property of avoiding a catastrophic
  outcome given a system fails to operate
  correctly. 
• Recoverability The ease for which a failed
  system can be restored to operational use. 

• We use a process similar to the development of
  the BMK in Caffall thesis to assure completeness
  of the model
• BMK development analyzed assertions to assure
  they addressed the seven dependability criteria
  (section 8H-Analysis of Prototype)
• CxSI used these criteria against the model
  instead of against assertions

11
Sample supporting analysis to ensure
Dependability in our model

• CxSI Dependability analysis required supporting
  analysis
• anytime/constant services analysis
• operational environment
• hazard analysis
• End to end use case views further increased model
  dependability
• Safety
• Consistency
• Recoverability
• Reliability

Anytime failure/continuous services analysis as a
separate analysis (availability and robustness)
Hazard analysis performed supports model
dependability (robustness).
12
Sample end-to-end analysis for Dependability in
our model
Use Cases
Parsed MSS Steps (with extensions)
End-to-End Summaries facilitate availability,
consistency, reliability, safety, recoverability
analysis
End-to End Filter criteria

• Level 2 MSS Steps
• Assertions
• Extension Conditions
• Extension Handling Steps (with frequency of
  occurrence)
• Action verbs for each Actor (with frequency of
  occurrence)
• Objects that get acted on (with frequency of
  occurrence)

13
Necessary Model Elaboration against L2
Requirements

• An assessment of the L2 requirements was made to
  determine need for elaboration of the model
• CxSI emphasis to have consistent level of model
  abstraction
• Level 1 independent of architecture elements,
  design reference missions
• Level 2 Driven by architecture and associated
  behavior provided in system ops-con
• Particularly, elaboration needs for Moon base and
  Mars necessary to determine need for Level 2
  elaboration of future system goals

Assessment resulted in some, (but not many),
elaborations to our Level 1 use cases and our
Level 2 Launch model
14
Review of SRM

• Peer Review
• In addition to our internal CxSI reviews, peer
  reviews are held when we complete major portions
  of our model, further increasing model confidence
• The peer review includes a combination of domain
  and modeling experts from the facility to review
  our products
• Feedback has been great and has resulted in
  better model and processes ?
• Project Review CxSI is targeting early August
  for CxP SEI review

RIDs State Machine
Formal schedule used for review
A PITS database is setup to track fixes we need
to incorporate
15
How do we validate the CxP Level 2 Requirements?

• Validation Plans
• Validation Generation Process and Status
• Validation Analysis Tasks
• SRM (Behavior, Interface)
• Requirements Trace
• Requirements Evaluation
• Other (Safety, Performance, Design)

16
Generation of CxSI Validation Plans

• In preparation for validation analysis, we
  performed a classification of CxP Level 2
  requirements into the following categories
• Behavior requirements that are driven from the
  behavior of the system,
• Interface requirements that are driven from
  interface interactions, or necessary for
  interfacing across the systems
• Performance requirements that indicate
  thresholds of goodness
• Design Implementation - trade based requirements
  could drive lower level behavior, but at Level
  2, are driven from the design
• Safety
• The validation plans are tailored to these
  categories
• e.g. SRM is used to validate the behavior and
  interface requirements but the Loss of Mission
  requirement stems from trades and has a
  performance aspect
• Our process to generate CxSI validation plans
  involved
• Initial Draft
• blue sky view of what needed to be done for
  requirements validation
• Reconciliation of this view with Facility WBS
• Team Peer Review (CxSI Team and JWST-Vorndran)
• (update) Infuse CxP processes into our assessment
  (both as information and as a standard)

17
CxSI Validation Analysis Tasks are binned into
four Categories
CxSI Validation Analysis Tasks
2. Requirements Trace Related Tasks ? for all
requirements C3. Level 1 to Level 2 trace
(verify project trace) C4. Level 2 to Level 3
trace (verify project trace) C5. Artifact
consistency verify CARD Section 3.7 is the same
as associated SRD Section 3.2
1. SRM Related Tasks ? for behavior and
interface Requirements C1. Validate Behavior
Requirements and Interface Interactions against
SRM C2. Interface Requirements (trades, and
trace related activities)
3. Requirements Evaluation Tasks ? for all
requirements C6. Requirements Evaluation per
requirement C7. Requirements Evaluation per
mission goal C8. Compare mission goal
requirements for completeness

• 4. Other Tasks
• C9. Safety Requirements ? for safety
  requirements
• C10. Design Implementation ? for design/trade
  related requirements
• C11. Performance Requirements ? for performance
  related requirements

Note this framework will be expanded in future
slides. Task IDs are used in planning, e.g. C1,
C2, etc
18
CxSI IVV Analysis Utilized multiple tasks,
tools, methods to increase assurance of coverage
19
SRM Related Validation Tasks

• SRM used to validate behavior requirements and
  interface interactions. Issues from analysis
  require CxSI assurance model is correct

Specific Plans Evaluate attributes of system
behavior extracted from behavior model to ensure
complete coverage of requirements at a consistent
level of abstraction. Perform a detailed
correlation between the SRM and behavior
requirements from the CARD. This trace must occur
in a two way manner 1) that the model represents
all behavior requirements in the CARD, and that
the CARD Requirements are correct (note if SRM
needs to be updated, CxSI process will be to find
reference for the change from somewhere else
besides the requirements or reference behavior as
an assumption) 2) that all behaviors (given level
of abstraction) are represented in the
requirements In performing the correlation, all
fields of the UC/AD should be evaluated for
completeness in the requirements. In terms of a
more detailed representation of behavior in the
IVV model than the CARD requirements, IVV
should either a) write an issue to capture the
missing behavior even though it may be in a lower
level document, or b) go into the lower level
requirements to determine if the behavior is
captured appropriately, and evaluate if a (lower
level) TIM should be generated which recommends
moving a requirement from Level 3/4 to Level 2.
20
Validation Data Capture Products
CxSI IVV SRM
CARD Requirements

• Updates to both the model and the CARD are
  expected
• Validation results will correspond to a snapshot
  of these products
• Results must be captured in a way that supports
  assessing change impacts and re-validation

21
Requirements to Model Correlation Access
Database (note still under construction)

• Step 1 Perform Correlation
• Database contains the requirements and the model
• Analyst links requirements to the elements of the
  model

• Step 2 Perform Validation Analysis
• Database reports enhance analysis by showing
  created links
• Analyst performs correctness and completeness
  analysis between requirements and model
• Discrepancies analyzed and addressed (TIMs)

CARD Requirements
CxSI Model (shown is a Use Case)
Output Report Correlation of the CxSI model
to CARD Level 2 requirement
Link Capability

• Database construction
• Analysis reports going through upgrade to include
  requirement/model text and means to capture
  analysis
• Widows and Orphans (requirements or model)
  reports
• Merge/replace with Facility tool when appropriate

The CxSI access database facilitates our model
validation analysis
22
Requirements Trace Validation Tasks

• Same as trace work performed under prior WBS 3.1
• CxSI will perform trace from Level 1?2 and Level
  2?3
• At level 3, additional consistency check
  necessary across artifacts

Specific Plans For provided trace, use parent
--gt child trace as primary artifact 1) ensure
appropriate links exist 2) ensure that the
linked items at the lower level constitute a
correct and complete decomposition of the upper
level requirements and 3) review widows/orphans
as a potential source of issues. 4) ensure that
child-parent and parent-child traces show
equivalent information In analysis
setup, Reconcile results with Civil Servant trace
performed. Ensure rationale explanations do not
contradict the trace or are necessary to complete
trace
23
Requirements Evaluation Related Validation Tasks

• Same as requirements evaluation work performed
  under prior WBS 3.2
• CxSI will perform evaluations in three areas
• On a requirement by requirement basis
  (correctness)
• Within a mission goal (correctness)
• Across mission goals (consistency)

• Specific Plans
• per requirement Is this requirement atomic
  (single thought), precise (unambiguous),
  testable. E.g. a good requirement (so if you were
  doing a design of it, you'd know what was
  expected) - and if not why (see Req't Eval
  Worksheet) lt-- note that the specific criteria
  may evolve
• Additionally, check that the requirement and
  rationale are consistent (this is a lower level
  issue). Ensure that information in the rationale
  should not be in the requirement (this would be a
  higher level completeness issue)
• evaluate requirements by mission goal (ISS,
  lunar, Mars) to ensure that multiple
  interpretations can't occur across suites of
  requirements or that contradictions don't exist
  within requirements. This check is to ensure
  consistency within a mission.
• Correlate related requirements for each mission
  against each other to ensure consistent level of
  depth across the four missions. ISS, moon
  sortie/base and Mars

24
Other (Safety, Design, Performance) Validation
Tasks

• Performed under prior WBS 3.2
• Combination of
• Tracing to standards/policies
• Identifying underlying trade or rationale
• Assessing technology readiness
• Domain Understanding

Specific Plans Safety Shuttle and ISS safety
requirements flow from some Program standard
documents, which are probably related to NASA
standards. CxP should have something analogous
(if they don't, we should use shuttle/ISS
standards). The analysis performed for the Level
2 requirements involved adherence to
Constellation safety standards. Safety
requirements are evaluated against the identified
aspects/criteria (Caffall) to assess the
requirements in terms of system-software safety
Fault Avoidance, Fault Warning, Fault Correction,
Fault Tolerance, Fail Operational, Fail Safe
Note that some of the safety requirements are
references to other documents/standards. As part
of this activity, a decomposition of the
referenced documents is likely necessary for
verification. Also, some of the safety
requirements will overlap and be also mapped into
performance, behavior or design
requirements Design Implementation Requirements
Analyze design implementation requirements to
ensure they are appropriate constraints on system
behavior at this level for subsequent
decomposition. Provide rationale for why design
based requirements are appropriate, e.g. traced
to a trade study. If requirements can't be traced
to a trade study, then the requirement is an
assumption by IVV and will be carried as
such. Performance Requirements. For each
performance requirement, analyze to ensure the
requirement is appropriate in defining the system
goal. Appropriateness is viewed in two ways 1)
The applicability of the requirement to meet the
overall performance objective (performed via
analysis) 2) The reasonableness of this
requirement from a technology readiness point of
view (for CxSI safety performance, extrapolate
these requirements from Shuttle/ISS safety)
25
Where We Are Now

• Finalized report with all analysis results
• C1, C8 Missing requirements
• C6, C10 Process problems manifest as technical
  problems
• C1, C2 SRM ? Requirement level of detail
• C10 OM Requirements bin
• C3, C4, C5 Requirement traceability/consistency
  
• C6 Requirement clarity vs. requirement
  rationale vs. requirement verification
• Next Steps
• Continue C1 with L3 requirements
• Update all analysis with future CARD/SRD releases
• More detailed scrutiny on the interfaces
• Use Validation results to focus future efforts