Hany H. Ammar

1
??? ???? ?????? ?????? ????? ??? ? ???????
??????? ??? ???? ????
Introduction to Risk Management And Software
Architecture Risk Assessment

• Hany H. Ammar
• LANE Department of Computer Science and
  Electrical EngineeringWest Virginia University,
  Morgantown, West Virginia, USA, and
• Faculty of Computers and Information, Cairo
  University, Cairo, Egypt

2
OUTLINE

• Risk Management
• Software Architecture Risk Assessment
• Maintainability-based risk
• Conclusions
• Next Steps

3
Risk Management
4
(No Transcript)
5
For NASA Programs

• RISK MANAGEMENT An organized, systematic
  decision-making process that efficiently
  identifies risks, assesses or analyzes risks, and
  effectively reduces or eliminates risks to
  achieving program goals.
• RISK A Program Risk is any circumstance or
  situation that poses a threat to crew or vehicle
  safety, Program controlled cost Program
  controlled schedule or major mission objectives,
  and for which an acceptable resolution is deemed
  unlikely without a focused management effort

6
Risk Management Cycle Identify Identify that a
risk exits and give it a meaningful
name. Analyze Determine the severity of the risk
according to the risk matrix. If the risk is
negligible (low to medium severity, low
likelihood of occurrence), stop here. However, if
the risk could cause damage to the system or the
system's users, continue. Plan Decide how to
combat the risk based on the risk's severity and
likelihood of occurrence. Mitigate Follow the
plan formulated in the previous phase as closely
as possible to combat the risk. If this approach
does not work, return to the previous phase and
make a new plan. If the plan does work, continue
analyzing the risk to determine whether it has
been reduced to an acceptable severity
level. Track Once the risk has been mitigated to
an acceptable severity level, the risk should be
tracked to ensure the continued control of the
risk. If at any time the risk seems to resurface,
the risk management cycle should begin again,
starting with the analysis phase.
7
(No Transcript)
8
Risk Definition

• According to NASA Software Safety Technical
  Standard, risk is defined as
• exposure to the chance of injury or loss. It is
  a function of the possible frequency of
  occurrence of the undesired event, of the
  potential severity of resulting consequences, and
  of the uncertainties associated with the
  frequency and severity.
• For software intensive systems, a risk is a
  combination of a likelihood of occurrence of an
  abnormal event or failure and the potential
  consequences or severity of that event or failure
  to a system's operators, users, or environment

9
Risk Matrix
Severity Likelihood of Occurrence Likelihood of Occurrence Likelihood of Occurrence Likelihood of Occurrence
Severity Probable Occasional Remote Improbable
Catastrophic High High High-Medium Medium
Critical High High-Medium Medium Medium-Low
Marginal High-Medium Medium Medium-Low Low
Negligible Medium Medium-Low Low Low
10
(No Transcript)
11
NASA IVV Facility

• NPD 2820.1C for Software IVV Policy
  states"Task the IVV Facility in Fairmont,
  West Virginia to manage the performance of all
  IVV for software identified per the established
  criteria, and for any other safety critical
  software (as defined in NASA-STD-8719.13)"

12
IVV Function

• Software Independent Verification Validation
  (IVV) is a systems engineering process employing
  rigorous methodologies for evaluating the
  correctness and quality of the software product
  throughout the software life cycle.
• Software IVV is adapted to the characteristics
  of the project. Different projects require
  different level of IVV

13
IVV Lifecycle Activities
System Requirements Review
Preliminary Design Review
System Test
Mission Readiness Review
System Retirement
Critical Design Review
S/W FQT
Launch
Initial IVVP Signed
Baseline IVVP Signed

• - IVV provides support and reports for Project
  milestones
• Technical Analysis Reports document major phases
• IVVP is updated to match changes in Project

IVV Provides CoFR
IVV Final Report
Concept Phase 2.0
Requirements Phase 3.0
Design Phase 4.0
Implementation Phase 5.0
Test Phase 6.0
Operations Maintenance Phase 7.0
IVV Phase Independent Support 1.0
Note numbers correspond to IVV WBS

• Life-cycle IVV is designed to mesh with the
  Project schedule and provide timely inputs to
  mitigate risk
• Dialog between the IVV Facility and the Project
  must begin before SRR
• For most Projects, IVV ends (and the Final
  Report is delivered) on or about MRR. Some
  Projects have extended S/W development
  post-launch or major upgrades/maintenance (e.g.
  Shuttle, MER)

14
Software Project Resolution
Project Resolution is commonly categorized into
three resolution types

• Successful Projects
• Completed and operational, and
• On Schedule
• On Cost
• With all originally specified features and
  functions
• Challenged Projects
• Completed and operational, but
• Behind Schedule-------? Project Risk
• Over Cost---------------? Project Risk
• With fewer features and functions than originally
  specified
• ------------? Product Risk
• Failed Projects
• Cancelled before completion or never implemented

15
Software CHAOS

• The Standish Group has examined 30,000 Software
  Projects in the US since 1994. This "CHAOS"
  research has revealed a decided improvement in IT
  project management with the implementation of
  standards and practices such as IVV. This
  improvement correlates with the rise in project
  success depicted in the chart below

Project Resolution History (1994-2000)
The Standish Group International, Inc. Extreme
CHAOS (2001) - The 2001 update to the CHAOS
report. http//www.standishgroup.com/sample_resear
ch/PDFpages/extreme_chaos.pdf
16
Error Detection/Correction

• Early error detection and correction are vital.
  The cost to correct software errors multiplies
  during the software development lifecycle. Early
  error detection and correction reduce costs and
  save time.

Direct Return on Investment of Software
Independent Verification and Validation
Methodology and Initial Case Studies, James B.
Dabney and Gary Barber, Assurance
Technology Symposium, 5 June 2003.
17
IVV Characteristics

• Includes Risk Identification and Mitigation
  Techniques
• Provides Independent Evaluation / Assessment of
• Are we building the product right? Verification
  
• Are we building the right product? Validation
• Requires Technical, Managerial and Financial
  Independence
• Makes a value added contribution, everyone shares
  the same mission success objective
• For NASA Management - Provides Mission Assurance
• For Project Management - Provides Unbiased Source
  of Help
• Helps deliver
• Risk Identification and Mitigation
• Increased Quality and Safety
• Improved Timeliness and Reliability
• Reduced Rework Cost
• NPD 8730.4 Requires NASA programs and projects
  that contain mission or safety critical software
  to document decisions concerning the use of IVV.

18
OUTLINE

• Risk Management
• Software Architecture Risk Assessment
• Reliability-based risk
• Performance-based risk
• Maintainability-based risk
• Component Ranking
• Conclusions
• Next Steps

19
Software Architecture Risk Assessment

• This work is funded in part by grants to West
  Virginia University Research Corp. from the NSF
  (ITR) Program, and from the NASA Office of Safety
  and Mission Assurance (OSMA) Software Assurance
  Research Program (SARP) managed through the NASA
  Independent Verification and Validation (IVV)
  Facility, Fairmont

20
Project Overview

• Risk Assessment of software architecture
  components, usage scenarios, and requirements
• Risk definition is based on
• Frequency of abnormal events
• Severity or consequences of events
• Reliability-based risk,
• Performance-based risk
• Requirementbased risk
• Severity Analysis
• Maintainability-based risk

What keeps satellites working 24/7 ?
21
Software Architecture Risk Assessment

• An architecture-based approach for risk
  assessment
• Components\connectors, requirements, and
  scenario risk,
• Define several types of risk factors
• Reliability-based Risk IEEE Trans. on Rel
  2001, on SE, 2002, 2003
• Probability of failure
• Severity or Consequences of this failure
• Maintainability-based Risk RAMS 06, ICSM 05,
  ICSM 04
• Probability of performing maintenance task
• Cost of performing this task
• The losses caused by low system maintainability
  can be
• High cost of maintenance effort
• Loss of the system by aging
• Performance-based Risk IEEE Trans. SE, Jan.
  2005
• Probability of missing timing or performance
  requirements
• Severity or Consequences

22
Importance / Benefits

• Identify the high risk components of the system
  in terms of Reliability/Maintainability/Performanc
  e

23
Software Architecture Risk Assessment Methodology

Requirements Model
System Architecture Model
Components Ranking
Components Risk Factors
24
OUTLINE

• Risk Management
• Software Architectures
• Software Architecture Risk Assessment
• Maintainability-based risk
• Conclusions
• Next Steps

25
Maintainability-based Risk

• ICSM 05 AbdelMoez, Goseva, Ammar, Mili,
  Fuhrman, Architectural level Maintainability
  Based Risk Assessment
• RAMS 06 AbdelMoez, Goseva, Ammar, Methodology
  for Maintainability Based Risk Assessment, Jan
  2006.

26
Importance / BenefitsMaintainability-based Risk

• According to Pigoski, 60-80 of the system
  budget is spent on maintenance

• Enhancements (perfective/ adaptive maintenance)
  account for 78-83 of the maintainer effort

27

Importance / BenefitsMaintainability-based Risk

• Unisys holds the NASA contract to maintain and
  support 14 million lines of ground software for
  the space shuttle
• There were 3,800 requirement changes made to the
  software after the loss of Challenger. These
  changes resulted in 900 software releases, of
  which 30 applied to the mission-control center
  with 3 of these being major upgrades

• Reference
• IEEE Software, Vol.6,  No.1,  pp. 116-119

http//hebb.cis.uoguelph.ca/dave/343/L
ectures/introduction.html
28
Importance / BenefitsTrade off Analysis for
Perfective Maintenance
Risk Factor
Components Risk
Components
Patterns
Maintainability risk for perfective maintenance
(open source case study Borg)
29
Software Architecture Risk Assessment
Methodology Maintainability-based Risk
SW Architecture
Requirments maturity Index / change / error
reports
(3) Estimate Size of Change (SC)
(1) Estimate components Initial Change
Probability (ICP)
(2) Estimate Change Propagation (CP) probabilities
ICPicpi
SCsci/j
CPcpi/j
(4) Estimate component risk factor
30
Maintenance change propagation
Outgoing maintenance
Incoming maintenance
31
Estimating Change Propagation
V1
C2
C1
1
V11
V12
V13
Change
Change in Provided Service
0

• Change Propagation Probabilities matrix CPcpij
  
• cpij is the probability that a change in Ci
  due to corrective/ perfective maintenance
  requires a change in Cj while maintaining the
  overall function of a system S
• cpij P(Cj ? Cj' Ci ? Ci' S
  S' )
• cpij is estimated by

32
Estimating Size of Change

• Size of change SCscij
• scij is defined as the ratio between the
  number of affected methods of the receiving
  component caused by the changes in the interface
  elements of the providing components and the
  total number of methods in the receiving
  component
• scij is estimated by

33
CM1 Maintainability-Based Risk in Adaptive
Maintenance Context
34
Case Study NASA CM1 UML Model Structure Diagram

• The UML-RT Model of CM1 was
• Developed by WVU students (Nathan, Tom and
  Rajesh, Summer 2004) based on the CM1 software
  design specification

35
Change Propagation Probabilities for CM1

• The Change Propagation probabilities CP is
  estimated using the CM1 UML model
• The Change Propagation probabilities CP can be
  automatically estimated from UML-RT models, or
  java source

36
Size of Change for CM1

• The Size of Change metrics SC is estimated using
  the CM1 UML model
• The Size of Change metrics SC Probabilities CP
  can be automatically estimated from UML-RT
  models, or Java source

37
Software Architecture Risk Assessment
Methodology Maintainability-based Risk
SW Architecture
Requirments maturity Index / change / error
reports
(3) Estimate Size of Change (SC)
(1) Estimate components Initial Change
Probability (ICP)
(2) Estimate Change Propagation (CP) probabilities
ICPicpi
SCsci/j
CPcpi/j
(4) Estimate component risk factor
38
Maintainability-based Risk For corrective
maintenance (case study CM1)
ICP is estimated using error reports data
39
Prioritizing Corrective Maintenance Tasks for CM1
40
Maintainability-based Risk Maintainability risk
for Adaptive maintenance (case study CM1)
ICP is estimated using change reports data
41
Tool Support
42
Technology Readiness Level The Software
Architecture Risk Assessment Tool Support
43
The tool will be developed as a web application
44
Conclusions

• Risk Management is vital to the success of
  projects and products
• A risk Assessment process is needed
• Software Architecture is a major determinant of
  software quality
• Software Architecture can be used to manage
  project and product risks
• Development of a methodology and a process for
  software architecture risk assessment
• Continued development of a software architecture
  risk assessment tool to support the methodology

45
Papers Published

• Vittorio Cortellessa, Katerina Goseva-Popstojanova
  , Kalaivani Appukkutty, Ajith R. Guedem, Ahmed
  Hassan, Rania Elnaggar, Walid Abdelmoez, Hany H.
  Ammar, Model-Based Performance Risk Analysis,
  IEEE Transactions on Software Engineering,
  January 2005, (Vol. 31, No. 1), pp.3-20.
• Katerina Goseva-Popstojanova, Ahmed Hassan, Ajith
  Guedem, Walid Abdelmoez, Diaa Eldin M. Nassar,
  Hany Ammar, Ali Mili, "Architectural-Level Risk
  Analysis Using UML", IEEE Transactions on
  Software Engineering, October 2003 (Vol. 29, No.
  10), pp.946-960.
• S. Yacoub, H. Ammar, A Methodology for
  Architectural-Level Reliability Risk Analysis,
  IEEE Transactions on Software Engineering, Vol.
  28, No. 6, June 2002.
• W. AbdelMoez, K. Goseva-Popstojanova, H.H.
  Ammar, Methodology for Maintainability-Based
  Risk Assessment, Proc. of the 52nd Annual
  Reliability Maintainability Symposium (RAMS
  2006), Newport Beach, Ca., January 23-26, 2006.
   
• Israr P. Shaik , W. Abdelmoez, R. Gunnalan, M.
  Shereshevsky, A. Zeid, H.H. Ammar, A. Mili, C.
  Fuhrman, Change Propagation for Assessing Design
  Quality of Software Architectures, Proc. of 5th
  IEEE/IFIP Working Conference on Software
  Architecture (WICSA), Pittsburgh, Pa., USA,
  November 6-9, 2005.
• AbdelMoez, W., I. Shaik, R. Gunnalan, M.
  Shereshevsky, K. Goseva-Popstojanova, H.H. Ammar,
  A. Mili, C. Fuhrman, Architectural level
  Maintainability Based Risk Assessment, Proc. of
  poster papers in IEEE International Conference on
  Software Maintenance (ICSM 2005), Budapest,
  Hungary, September 25-30,2005.
• W. Abdelmoez, D. M. Nassar, M. Shereshevsky, N.
  Gradetsky, R. Gunnalanm and H. H. Ammar, Bo Yu,
  and Ali Mili "Error Propagation in Software
  Architectures". In Proceedings of the 10th
  International Symposium on Software Metrics
  (METRICS'04), September 11 - 17, 2004 , IEEE
  Comp. Soc., pp 384-393
• Abdelmoez, W., M. Shereshevsky, R. Gunnalan, H.
  H. Ammar, Bo Yu, S. Bogazzi, M. Korkmaz, A. Mili,
  "Software Architectures Change Propagation Tool
  (SACPT), 20th IEEE International Conference on
  Software Maintenance (ICSM'04) September 11 -
  14, 2004 , Chicago, Illinois, IEEE Comp. Soc., pp
  517
• A. Hassan, K. Goseva-Popstojanova, and H. Ammar,
  UML Based Severity Analysis Methodology,
  Proceedings of the 2005 Annual Reliability and
  Maintainability Symposium (RAMS 2005),
  Alexandria, VA, January 2005.