Module 1

1
Module 1

• Welcome

2
Overview

• Introductions
• Administrative
• Course Objectives
• Course Schedule
• Style of Course
• Course Materials

3
Administrative

• Breaks
• Sign-In Sheet

4
Targeted Audience

• Mix of project personnel with variable levels of
  experience in KSC development projects
• Prerequisites
• Project management or systems engineering
  experience (at least one year)
• Assumptions
• Prior knowledge of risk or risk management is not
  required

5
Course Objectives

• Understand the concepts and principles of
  Continuous Risk Management and how to apply them
• Develop basic risk management skills for each
  component of Continuous Risk Management
• Be aware of key methods and tools
• Understand how CRM could be tailored to a project
• Be able to differentiate between Risks and
  Problems

6
Module 2

• Introduction

7
Overview

• Agency Risk Management (RM) Requirements
• Risk Definitions
• RM/Project Management Relationship
• Risk Management Benefits
• Continuous Risk Management (CRM) Process
• CRM Process Application
• Risk Management Planning/Documentation
• Who Performs Continuous Risk Management?

8
Agency Risk Management Requirements

• Risk Management Documentation
• NPD 7120.4, Program/Project Management, describes
  the management systems for program/project
  formulation, implementation, and evaluation
• NPG 7120.5, NASA Program and Project Management
  Processes and Requirements, dictates basic risk
  management requirements
• NPG 8000.4, Risk Management Procedures and
  Guidelines, provides additional information for
  applying risk management to programs and projects
  as required by NPG 7120.5
• Procurement Notice (PN) 97-58, Risk Management

9
Agency Risk Management Requirements

• Fundamental Risk Management Requirements
• Program and project decisions shall be made on
  the basis of an orderly risk management effort
• Risk management includes identification,
  assessment, mitigation, and disposition of risk
  throughout the project formulation, approval,
  implementation, and disposal phases
• Project/Program Manager (PM) has the overall
  responsibility for the implementation of risk
  management, ensuring an integrated, coherent risk
  management approach throughout the project

10
Agency Risk Management Requirements

• Fundamental Risk Management Requirements
• Risk management planning will be developed during
  the project/program formulation phase, included
  in project/program plans, and executed during the
  implementation phase
• Programs and projects will develop and maintain
  prioritized risk lists
• Programs and projects must develop and clearly
  communicate success criteria to all levels of
  the program and project to define the scope of
  the effort and to guide risk decisions

11
Agency Risk Management Requirements

• Fundamental Risk Management Requirements
• Programs and projects must define, within the
  constraints imposed upon them (e.g., budget,
  schedule), what level of risk (i.e., acceptable
  risk) is reasonable for the program/ project
  to accept while still achieving mission success
• Primary risks (i.e., risks having high
  probability and high impact/severity) must be
  classified, with acceptance rationale documented
  and concurred with by the Governing Program
  Management Council (GPMC)

12
Risk Definitions

• Risk is the combination of the probability
  (qualitative or quantitative) that a program or
  project will experience an undesired event (cost
  overrun, schedule slip, safety mishap, security
  compromise) and the consequences (impact) of the
  undesired event, were it to occur.
• NPG 8000.4

13
Risk Definitions
Risk involves the impact of the event should it
occur.
14
Some Perspectives on Risk

• Risk will always be present in programs and
  projects
• Not all risk can be avoided
• Management and stakeholders must be active
  participants in the mission risk acceptance
  process
• Risks are different from problems

15
Goal of Risk Management

• Achieving Mission Success
• Provide program/project managers principles and
  practices for decision making
• Search out, identify, and manage risk
• Keep safety paramount
• Deliver a quality product on time and within cost

16
Success Criteria Emphasis

• Program/project teams must develop clear Success
  Criteria during the formulation phase
• Success criteria must be clearly communicated to
  all levels of the program and project to define
  scope of the effort and to guide risk decisions
• Success criteria need to be developed at system,
  subsystem, and component level to define trade
  space and support risk decisions
• Success criteria will continue to evolve
  throughout the life cycle of the project

17
Risk Management/Project Management Relationship
Project Management
18
Risk Management Benefits

• Early identification of potential risks
• Facilitates setting priorities
• Increases chance of project success
• Enables more efficient use of resources
• Promote teamwork by involving personnel in all
  levels of the project
• Information for tradeoffs is based on priorities
  and quantified assessments
• Identify hidden risks

19
Everybody Wants to Understand Risk
20
Continuous Risk Management Process

• Continuous Risk Management (CRM) is a structured,
  formalized management practice with processes,
  methods, and tools for managing risks in a
  project
• It provides a disciplined environment for
  proactive decision making to
• Assessment (continual) of what could go wrong
  (risks)
• Determination of which risks are most important
  to deal with
• Implementation of mitigation strategies to deal
  with those risks
• Assured, measured effectiveness of the
  implemented mitigation strategies

21
Continuous Risk Management Process
22
CRM Process Components

• Identify
• Search for and locate risks before they become
  problems
• Analyze
• Convert risk data into useable information for
  determining priorities and making decisions
• Plan
• Translate risk information into planning
  decisions and mitigating actions (both present
  and future), and implement those actions

23
CRM Process Components

• Track
• Monitor risk indicators and mitigation actions
• Control
• Correct risk mitigation plans deviations and
  decide on future actions
• Communicate and Document
• Provide information to project on risk activities
  and current/future risks, and emerging risks

24
Relationship Among CRM Functions

• Throughout the project life cycle, risk
  components evolve
• Continuously
• Concurrently
• Iteratively

25
Risk Management Data Flow
Statements of risk
Project goals
Context
Resources
and constraints
Impact
Probability
Statements of risk
Timeframe
Individual
Context
Classification
uncertainties
Rank
Classification
Class 1
Class 2
Risk
Risk
Identify
Analyze
Plan
Risk
Risk
Risk
Class 3
Master list
of risks
Risk
Risk
List of risks
Project
Group/team
Top
uncertainties
data
N
Statement of risk
Decisions
Context
Status reports
replan
Impact
Probability
close
risks
invoke
Timeframe
mitigation
contingency
Classification
plans
Resources
continue
Rank
tracking
Plan Approach
Statements of risk
Context
Statements of Risk
Track
Control
Impact
Plan
Action plans
Probability
Context
Impact
Timeframe
Probability
Classification
Timeframe
Rank
Classification
Plan Approach
Project
Rank
Risk mitigation
Project
Status
Plan Approach
plan measure
data
data
Status
Control Decision
26
Continuous Risk Management Application
27
When Should CRM be done?
28
RM Planning/Documentation

• Risk Management planning early in the project
  life cycle (i.e., formulation) is required per
  NPG 7120.5 (Section 4.3.2a) NPG 8000.4
• Options
• Stand-Alone Risk Management Plan (Medium-to-Large
  Projects)
• Risk Management Section in Project Plan (Smaller
  Projects)

29
Risk Management Plan Details

• Purpose
• Documents the risk management practice
  (processes, methods, and tools) to be used for a
  specific project
• Contents
• Introduction/Overview
• Project organization, roles, responsibilities
• Practice details (e.g., how risks are identified)
• Risk management milestones (e.g., quarterly risk
  list reviews)
• Risk information documentation (e.g., database)
• De-scope options
• Available Information
• SET/YA-B, conjunction with SHIA/QA-C, has
  established risk management and project plan
  templates, and can offer consulting and guidance
  during plan development

30
Relationship to Everyday Practice

• Learning
• Continuous Risk Management is similar to
  incorporating
• any new habit
• into your daily life.

31
Core Risk Management Team
32
Who Performs Continuous Risk Management?

• Everyone!

33
Module 3

• Identify

Control
Identify
Track
Communicate Document
Analyze
Plan
34
Overview

• Identification activities
• Capturing statements of risk
• Capturing the context of a risk
• Identification methods and tools
• Brainstorming
• Questionnaires and checklists
• Personal knowledge/experience
• RM/SMA analysis tools (FMEA, FTA, PRA)
• Lessons Learned

35
Recording Data on the Risk Information Sheet

• Risk Information Sheet
• Fields to be Completed in Identification Phase
• ID
• Date Identified
• Risk statement
• Origin
• Risk Context

36
Capturing Statements of Risk

• Purpose
• Arrive at a concise description of risk, which
  can be understood by everyone and acted upon
• Description
• Involves considering and recording the condition
  that is causing concern for a potential loss to
  the project, followed by a brief description of
  the potential consequences of this condition

37
Components of a Risk Statement
there is a possibility that
Given the

Consequence
will occur
Condition
Risk Statement

• Condition A single phrase that identifies
  possible future problems, and describes current
  key circumstances and situations that are causing
  concern, doubt, anxiety, or uncertainty
• Consequence A single phrase or sentence that
  describes the key, negative outcome(s) of the
  current conditions

38
Elements of a Good Risk Statement

• Consider these questions when looking at a risk
  statement
• Is it clear and concise?
• Will most project members understand it?
• Is there a clear condition or source of concern?
• If a consequence is provided, is it clear?
• Is there only ONE condition, followed by one (or
  more) consequence?

39
Example Risk Statements

• Good or bad risk statements?
• Object Oriented Development (OOD)!
• The staff will need time and training to learn
  object oriented development.
• This is the first time that the software staff
  will use OOD the staff may have a lower than
  expected productivity rate and schedules may slip
  because of the associated learning curve.

40
Capturing the Context of a Risk

• Purpose
• Provide enough additional information about the
  risk to ensure that the original intent of the
  risk can be understood by other personnel,
  particularly after time has passed
• Description
• Capture additional information regarding the
  circumstances, events, and interrelationships not
  described in the statement of risk

41
Context of a Risk Statement

• An effective context captures the what, when,
  where, how, and why of the risk by describing the
  circumstances, contributing factors, and related
  issues (background and additional information
  that are NOT in the risk statement)

42
Elements of Good Context

• Consider these questions when looking at the
  context
• Can you identify which risk statement this
  context is associated with?
• Is it clear what the source or cause of the risk
  is?
• Is it clear what the impact might be?
• Would you know who to assign the risk to for
  mitigation? Would they (the person responsible
  for risk mitigation) know what to do?
• Would you be able to tell if the risk has gone
  away?

43
Example Context (1)

• Risk Statement
• This is the first time that the software staff
  will use Object Oriented Development (OOD) the
  staff may have a lower-than-expected productivity
  rate and schedules may slip because of the
  associated learning curve.
• Risk context
• Its a typical NASA project new concepts
  without training.
• Is this an example of good or bad context?

44
Example Context (2)

• Risk Statement
• This is the first time that the software staff
  will use OOD the staff may have a lower than
  expected productivity rate and schedules may slip
  because of the associated learning curve.
• Risk Context
• Object oriented development is a very different
  approach that requires special training. There
  will be a learning curve until the staff is up to
  speed. The time and resources must be built in
  for this or the schedule and budget will overrun.

45
Risks are Not Problems (1)

• Risk
• A future event
• A potential problem
• Has a level of uncertainty (gt0 and lt100 chance
  of occurrence)
• Problem
• Is happening now
• Must be dealt with immediately
• No uncertainty (its occurring now)

46
Risks are Not Problems (2)

• Risks are anticipated problems
• Example Delivery of Class S parts on schedule
  is questionable
• A Problem is a Risk that has occurred
• Example Class S parts have not been delivered
• Problems may create new risks
• Change in design, increased testing
• Schedule slip, screening cost

47
Brainstorming

• Purpose
• Group method for generating ideas
• Description
• Participants verbally identify ideas as they
  think of them, thus providing the opportunity for
  participants to build upon or spring off of ideas
  presented by others

48
Risk Statement Identification Tools --
Taxonomy-Based Questionnaire (TBQ)

• Taxonomy The classification of something in an
  ordered system that indicates natural
  relationships division into ordered groups or
  categories
• TBQs are questionnaires organized according to
  the taxonomy of a particular body of knowledge
• TBQs provide a structured approach for
  identifying risks associated with a project
• CRM Guidebook (pp. 471-493)

49
Additional Risk Identification Methods

• Failure Modes and Effects Analysis
• Fault Tree Analysis
• Probabilistic Risk Assessment (PRA)
• Lessons Learned (http//llis.nasa.gov)
• Various Other Checklists

50
Risk Identification Data Flow
Identify
Capture statement
of risk
Capture context of
risk
51
Risk Information Sheet(After the Identification
Phase)

• Risk Information Sheet (RIS)
• after the CRM Identify phase

52
Identification Phase Summary

• Good risk statements
• Contain ONLY one condition
• Contain at least one consequence
• Are clear and concise
• Good context
• Provides further information not in the risk
  statement
• Ensures that the original intent of the risk can
  be understood by other personnel, even after time
  has passed
• Communication is an integral part of risk
  identification

53
Module 4

• Analyze

Control
Identify
Track
Communicate Document
Analyze
Plan
54
Overview

• Analysis activities
• Evaluating attributes of risk
• Classifying risks
• Prioritizing risks

55
Recording Data on the Risk Information Sheet

• Risk Information Sheet
• Fields to be Completed in Analysis Phase
• Priority
• Probability
• Impact
• Timeframe
• Class

56
Evaluating Attributes of Risk

• Purpose
• To gain a better understanding of the risk by
  determining the expected impact, probability, and
  timeframe of the risk
• Description
• Involves establishing values for
• Impact the loss or effect on the project if the
  risk occurs
• Probability the likelihood the risk will occur
• Timeframe the period when you must take action
  to mitigate the risk (NOT when the risk will
  occur)

57
Levels of Analysis
58
Tri-Level Attribute Evaluation Example

• Each attribute has one of three values
• Impact catastrophic, critical, marginal
• Probability very likely, probable, improbable
• Timeframe near-term, mid-term, far-term
• Risk Exposure

Probability
Improbable
Probable
Very Likely
Catastrophic
Critical
Impact
Marginal
59
Example Probability Definitions

• A risk is very likely if there is a gt70
  probability that it will occur
• A risk is probable if there is a 30-70
  probability that it will occur
• A risk is improbable if there is a lt30
  probability that it will occur

60
Example NASA Safety Impact Definitions

• Catastrophic (Class I)
• Loss of entire system
• Loss of human life
• Permanent human disability
• Critical (Class II)
• Major system damage
• Severe injury
• Temporary disability

• Marginal (Class III)
• Minor system damage
• Minor injury (e.g., scratch)
• Negligible (Class IV)
• No system damage
• No injury (possibly some aggravation)

61
Example Impact Definitions
Catastrophic
Critical
Marginal
Schedule
gt 20
10 20
0 10
Slip
Cost
gt 15
5 15
0 5
Overrun
System is
Major
Data lost
Technical
lost
function
lost
62
Example Timeframe Definitions

• A risk has a near-term timeframe if the project
  must take mitigation action in the next 90 days
• A risk has a mid-term timeframe if the project
  must take mitigation action in the next 90-180
  days
• A risk has a far-term timeframe if the project
  does not need to take mitigation action within
  the next 180 days

63
Standardized Agency 5x5 Risk Matrix
64
Definition of a Primary Risk

• NPG 8000.4 defined a Primary Risk as those
  undesirable events (risks) having both high
  probability and high impact/severity
• Characterization of a Primary Risk as
  Acceptable shall be supported by rationale,
  with concurrence of the Governing Program
  Management Council (GPMC), that all reasonable
  mitigation options (within cost, schedule, and
  technical constraints) have been instituted

65
Risk Matrix Application
66
Example Impact Level Definitions
Impact Rating Safety Technical/ Performance Cost Schedule Programmatic
5 Catastrophic, may cause death or permanently disabling injury Cannot meet minimum success criteria gt15 Over Run Unrecoverable Project delay Forces project cancellation review
4 Critical, may cause severe injury or occupational illness Major impact to full mission success gt10 Over run Major slip in key milestone Major impact to budget, schedule, or mission success
3 Moderate, may cause injury or occupational illness Loss of system, With work-arounds, moderate impact on full mission success gt5 Over run Minor slip in need date Moderate impact to budget, schedule, or technical success of mission
2 Negligible, no adverse affect to personal safety or health Loss of redundancy or functional degradation, Minor impact to full mission success Reserves eroded Meet need date with no margin Can be covered by reserves, leaves no contingency for other Risks, minor impact to technical success
1 Meets safety requirements Component degrades, minor impact to full mission success Minimal - Begins to erode reserves Minimal - Begins to erode margins Minimal budget, schedule, or technical impact to project
67
Example Probability Rating Definitions
Probability Rating Probability of Occurrence(cost schedule ) Probability of Occurrence (performance) Safety Probability of Occurrence
5 81 gt99 51 gt99 10-6 gt_ P
4 61 80 31 50 10-3 gt_ P gt_ 10-6
3 41 60 15 30 10-2 gt_ P gt_ 10-3
2 21 40 5 14 10-1 gt_ P gt_ 10-2
1 0lt 20 0lt 4 10-6 P gt 10-1 gt_ P
68
Classifying Risks

• Purpose
• Look at a set of risks and how those risks relate
  to each other within a given structure
• Efficiently sort through large amounts of data
• Description
• Involves grouping risks based on shared
  characteristics. The groups or classes show
  relationships among the risks
• Example classifications
• Safety Management
• Cost Environmental
• Schedule Security (plus IT Security)
• Technical Performance Political

69
Prioritizing Risks

• Purpose
• Sort through a large amount of risks and
  determine which are most important
• Separate out which risks should be dealt with
  first (the vital few risks) when allocating
  resources
• Description
• Involves partitioning risks or groups of risks
  based on the Pareto vital few sense and ranking
  risks or sets of risks based upon a criterion or
  set of criteria
• Prioritization should utilize the projects
  success criteria
• Prioritization should yield the projects
  Primary Risks

70
Two Step Risk Prioritization
List of risks
Order the Top N risks
Prioritized Ordered Master List of
Top N RISKS
Master list
of risks
Select the top or N risks

Top 10 Top 20
71
Risk Prioritization Using Multivoting

• Multivoting is a technique for prioritizing a
  subset of items from a larger set (usually
  one-third of the items on a list)
• Each evaluator in a group gets a certain number
  of votes (e.g., 3) to use for risk prioritization
• Each evaluator then assigns a number between 1
  and 3 (i.e., the number of votes they have) to
  the risks they feel are most important (e.g., a
  3 is a higher priority risk than a 1)
• Totals for each risk are tallied highest
  priority risks are those with the most points

72
Multivoting Example
Example Risk order of criticality 5
participants H B A C J 12 risks 3 weighted
votes (1 2 3)
73
Risk Prioritization Using a Risk Assessment Code
(RAC)

• The use of Risk Assessment Code (RAC) is an
  additional, qualitative method for prioritizing
  risks
• The RAC is a tool used to express comparative
  risks in all categories by evaluating both the
  potential severity of a condition and the
  probability of its occurrence
• RACs can be utilized to determine a projects
  Primary Risks (i.e., those risks with both a
  high probability and a high impact) e.g., those
  risks with a RAC of 1 or 2
• RACs can be assigned in a tailored manner to meet
  the needs or complexity of a program or project

74
Risk Assessment Code (RAC)Project Example
Likelihood Estimate
Impact/Severity
High risk
Medium
Low

• RACs are assigned a number based on the risk
  exposure (product of probability times impact),
  with each attribute level having a score from 1
  to 5 in this risk matrix
• For this example, the RED (High Risk) risks
  having a value of 15-25 would be classified as
  Primary Risks

75
Risk Analysis Data Flow (1)

• Evaluate
• Impact (I)
• Probability (P)
• Timeframe (T)

• Classify
• Identify duplicates
• Consolidate risks to sets

• Prioritize
• Identify Pareto top N
• Rank top N
• Determine RAC

Risk I P T Risk a M M F Risk b M L
N Risk c L H N . . .
Consolidate risks
Risk I P T Risk set A H M F
----- ----- Risk b M L N Risk c
L H N . . .
Sort by evaluation results
Risk I P T Risk n H H
N Risk s H M N Risk set A H M F
----- ----- Risk c L H N
Top N 1. 2. 3. . . .
Rank order the Pareto top N
Pareto top N
RAC
76
Risk Analysis Data Flow (2)
Statement of risk
Context
77
Sample Risk Management Data Flow
78
Risk Information Sheet after Analyze Phase

• Risk Information Sheet (RIS)
• after the CRM Analyze phase

79
Analyze Phase Summary (1)

• Evaluate risks at a level that is sufficient to
  determine the relative importance
• Select attribute definitions (e.g., catastrophic
  impact, RAC 1) that make sense for your project
  document these in the project Risk Management
  Plan
• Classify risks to help the project understand the
  risks
• Group related risks into sets to help build more
  cost-effective mitigation plans

80
Analyze Phase Summary (2)

• Prioritize to determine which risks should be
  dealt with first when allocating resources
• Prioritize the risks based on the criteria for
  what is most important to the project
• Communication is central to
• Defining project evaluation definitions
• Evaluating risks
• Selecting a project classification scheme
• Classifying risks
• Defining prioritization criteria
• Identifying and prioritizing the Top N risks

81
Module 5

• Plan

Control
Identify
Track
Communicate Document
Analyze
Plan
82
Overview

• Risk Planning activities
• Assigning responsibility
• Determining risk mitigation approach
• Defining scope and actions
• Mitigating a set of related risks

83
What Is Risk Planning?

• Risk planning is the function of deciding what,
  if anything, should be done with a risk
• Risk planning answers the questions
• Who does planning?
• Is it my risk? (responsibility)
• What can I do? (approach)
• How much and what should I do? (scope and actions)

84
Recording Data on the Risk Information Sheet

• Risk Information Sheet
• Fields to be Completed in Risk Planning Function
• Assigned to
• Mitigation Strategy
• Contingency Plan and Trigger

85
Risk Planning Decision Flowchart
86
Project Considerations Regarding Risk Planning

• What are the risk attributes?
• Is it a Primary Risk?
• What is currently important to the project,
  management, customer, or user?
• Are there critical milestones the project is
  currently is facing?
• What limits or constraints do the project,
  organization, or manager have?
• What resources are available for mitigation?
• How does the risk fit into the overall project
  issues and concerns? When is the best time to
  address or mitigate a risk?

87
Assigning Responsibility for Risk Planning

• Purpose
• Ensure that no risks are ignored
• Make effective use of expertise and knowledge
  within the project when planning for risk
  mitigation
• Ensure that risks are being managed by those with
  the appropriate abilities, knowledge, and
  authority to commit resources for mitigation
• Description
• Involves reviewing the risk(s) and determining
  who is best able to deal with the risk(s)

88
Determining Risk Planning Approach

• Purpose
• Ensure you know enough to make an informed
  decision
• Pick an appropriate approach for effective
  management of the risk(s)
• Establish measurable mitigation goals that
  provide a target for evaluating success and
  direction during the development of action plans
• Description
• Involves reviewing the risk(s) and determining
  the best approach to take

89
Risk Planning Approaches
90
Contingency Planning

• Not all risk mitigation strategies can or should
  be carried out immediately, for example
• There may not be sufficient funding at this time
• Other circumstance (available personnel) may not
  be right
• Risk may be low probability, catastrophic impact
  with an expensive mitigation strategy
• Current strategy might not be working
• May need to develop a contingency risk mitigation
  strategy (to be used as Plan B if Plan A fails)
• Contingency risk mitigation strategy plans are
  held in reserve until specific trigger conditions
  are true or certain events occur
• Watch for the conditions and events!

91
Defining Scope and Actions for Risk Mitigation
Strategies

• Purpose
• Take a balanced approach in developing effective
  actions to mitigate risks
• Description
• Involves reviewing the risk(s) and determining
  the appropriate level of mitigation to take and
  the goal of the mitigation

92
Determining Risk Mitigation Goals and Success
Measures

• There is a need to set realistic, measurable (or
  verifiable) goals for mitigating individual
  risks
• Avoid changes to scheduled milestones
• Eliminate change requests unsupported by funding
  to implement the change
• Define mitigation success criteria it is
  important to know when youve succeeded or failed
  in mitigating risks
• All current change requests implemented by
  DD/MM/YY, with no change to scheduled milestones

93
Discussion -- Risk Mitigation Goals and Success
Measures

• Risk 7
• Science requirements have substantial TBDs late
  completion of TBDs likely, with reduction in
  adequate testing time, possible science
  application software failure, incorrect science
  data being captured, hardware damage if incorrect
  safety limits were provided, extensive rework and
  substantial cost overruns, mission failure if
  problems not found before system is in operation
• What risk mitigation goals and success measures
  would you look for?

94
Risk Planning Data Flow
Strategies Actions Triggers

Consequences may be added
to the risk statement if not
already documented
95
Risk Information Sheet(After the Risk Planning
Phase)

• Risk Information Sheet (RIS)
• after the CRM Risk Planning phase

96
Risk Information Sheet(After the Risk Planning
Phase)

• Risk Information Sheet (RIS)
• after the CRM Risk Planning phase

97
Risk Planning Phase Summary (1)

• The result of risk planning is a documented
  decision about what should be done with each risk
• The risk planning approach, as well as the
  mitigation strategy-related details are
  documented on the Risk Information Sheet. The
  risk planning approaches and their related risk
  planning documentation are
• Research (Research Planning)
• Accept (Acceptance Rationale)
• Watch (Tracking Requirements)
• Mitigate (Mitigation Strategy, Actions,
  Completion Dates, Triggers, Contingency
  Strategies)

98
Risk Planning Phase Summary (2)

• Mitigate unacceptable risks to the project
• You cant mitigate all risks but you need to
  understand which risks you are taking
• Watch the risks that you cant currently mitigate
  and dont want to accept
• Unassigned tasks tend to fall through the cracks
• Dont over plan documentation of mitigation
  strategy actions on the Risk Information Sheet is
  sufficient for almost all identified project risks

99
Module 6

• Track

Control
Identify
Track
Communicate Document
Analyze
Plan
100
Overview

• Tracking activities
• Acquiring Data
• Compiling and Evaluating Data
• Reporting Status (planning, risks)

101
What do We Mean by Tracking?

• Tracking
• A process for watched and mitigated risks where
  related data are acquired, compiled, analyzed,
  and reported
• Risks can be tracked individually or in sets

102
Recording Data on the Risk Information Sheet

• Risk Information Sheet
• Fields to be Completed or Updated in the Tracking
  Function
• Priority
• Probability
• Impact
• Timeframe
• Status
• Status Date

103
Tracking Risks and Mitigation Strategies

• Tracking risk mitigation strategies will indicate
• Whether the mitigation strategy is being executed
  correctly
• If risk mitigation is on schedule (including
  action items)
• Tracking individual risk attributes will indicate
• Mitigation strategy effectiveness
• Is the impact/probability reduced?

104
Risk Metrics

• Risk Metrics are used to
• Measure attributes of a risk
• Impact, probability, and timeframe
• Other risk- or project-specific attributes
• Provide meaningful information to enable more
  informed control decisions
• Assess the impact or success of a mitigation
  strategy
• Identify new risks (if indicated by the risk
  metrics)

105
Acquiring Data

• Purpose
• To collect tracking data for a given risk
• Description
• A process that includes all of the steps
  associated with collecting information about and
  updating the values of risk measures and status
  indicators for watched and mitigated risks

106
Considerations When Acquiring Data

• Status information is only as good as its
  accuracy and timeliness
• Stale data are more dangerous to decision makers
  than no data at all
• When a group of indicators is required, all of
  the data must be acquired from the same time
  period
• Collect just the data needed to track the
  projects risks. Collect only what you need and
  use what you collect

107
Data Acquisition (Metrics Examples)

• Requirements
• Ambiguity Weak Phrases and/or Optional Language
• Measure of Completeness TBD TBA TBR
• Design and Implementation
• Structure/Architecture Complexity and Size
• Testing
• Problem Reporting Tracking Open, Closed,
  Severity
• Defect Density
• Process
• Schedule Effort, Completion Rates
• Budget

108
Compiling and Evaluating Data

• Purpose
• Organize and understand the relevant tracking
  data for a given risk
• Description
• A process in which data for a given risk is
  combined, calculated, organized, and interpreted
  for the tracking of a risk and its associated
  mitigation strategy

109
Triggers/Thresholds

• A value of an indicator that specifies the level
  at which an action, such as implementing a
  contingency plan, may need to be taken
• Triggers and thresholds are generally used to
• Provide early warning of an impending critical
  event
• Indicate the need to implement a contingency plan
  to preempt a problem
• Request immediate attention for a risk
• Triggers and thresholds are effective if
• They do not trip unnecessarily
• They are easy to calculate and report

110
Example Use of Triggers
Over budget
acceptable
Within Budget
Under budget
Risk 100Project resources (personnel number and
availability) and schedules were underestimated
schedule slips, cost overruns, reduction in
adequacy of development processes (especially
testing time adequacy) likely.
111
Process (Metrics Example)

• Risk 6 Project software schedule and resources
  were underestimated schedule slips, reduction in
  adequate testing time
• Data to be collected
• Effort per activity
• Trigger
• Exceeds expected percentages

112
Process Metrics Example(Effort per Life Cycle
Phase)
Actual/Projected Effort
Test
18
Req/Design
Test
Req/Design
30
33
34
Current Status
Development
48
Development
37
Risk - Decrease in Testing projected
113
Reporting Data

• Purpose
• Communicate risk status reports to support
  effective decision making (inputs for the
  Control function)
• Description
• A process in which status information about risks
  and mitigation strategies is communicated to
  decision makers and team members

114
Reporting Considerations

• What information needs to be reported?
• What presentation formats best present the
  analyzed data?
• Does the information and the format of the report
  provide the basis needed by decision makers?

115
Sample Risk Management Data Flow
116
Standardized Agency 5x5 Risk Matrix
117
Top Risk List and Risk Matrix Example
118
Risk Focus Chart Example
119
Stoplight / Fever Chart
Condition/Change
Risk ID
Risk Statement
Assigned To
Planning Approach
Remaining Milestones
Comments
Yellow
14
Contracting different test facility insufficient
testing, damage. Science reqt substantial TBDs
late completion, incomplete testing, wrong
data. SW schedule and resources under estimated
schedule slips, cost overruns.
Green
7
Red
6

120
Reporting Schedule

• Reports are generally delivered as part of
  routine project management activities
• Weekly status meetings
• Monthly project meetings
• The frequency of reporting depends on
• The reporting requirements for each risk or risk
  set
• The manner in which the report will be used
• Exception reporting may be necessary

121
Risk Tracking Data Flow
122
Tracking Phase Summary

• Tracking reports communicate information required
  for effective control decisions
• Tracking information and reports can include
  quantitative indicator data as well as more
  subjective information (e.g., recommendations)
• Tracking information is not limited to formal
  reporting mechanisms
• Informal reporting of risk-related information by
  all project personnel can aid decision making
• Risk tracking should be integrated with standard
  management practices risk management should be
  tailored for a project

123
Module 7

• Control

Control
Identify
Communicate Document
Track
Analyze
Plan
124
Control Overview

• Control activities
• Evaluate tracking results
• Decide on course of action
• Execute control actions

125
What is Control?

• Control
• A process in which decisions are made based on
  the data presented in the tracking reports
• Risks can be controlled individually or in sets

126
What Is Effective Control?

• Monitoring the quality of risk mitigation
  execution
• Assessing the effectiveness of mitigation
  strategies
• Assessing significant changes in risks and trends
• Determining appropriate responses
• Executing the plan of attack
• Communicating the above information

127
Recording Data on the Risk Information Sheet

• Risk Information Sheet
• Fields to be Completed in Risk Control Function
• Approval
• Closing Date
• Closing Rationale
• Status

128
Evaluate Tracking Results

• Purpose
• Allows decision makers to identify significant
  changes in risks, to assess the effectiveness of
  mitigation strategies, and to accurately
  determine the best courses of action
• Description
• Uses tracking data to reassess project risks for
  trends, deviations, and anomalies

129
Metric Trend Analysis

• The Risk Management Plan can document which
  project metrics to track
• Trend and data analysis of project metrics can be
  used to identify new risks
• Trends can be observed through the evaluation of
  successive reports
• Persistent lateness in taking action
• Oscillating priority values
• Significant changes in the number of high impact
  risks or risks of a particular type

130
Trending Metrics Example

• Given concerns about unstable or incomplete
  requirements, which metrics might be useful in
  controlling this risk area?
• Risk 7 Science requirements have substantial
  TBDs late completion of TBDs likely, with
  reduction in adequate testing time, possible
  science application software failure, incorrect
  science data being captured, hardware damage if
  incorrect safety limits were provided, extensive
  rework and substantial cost overruns, mission
  failure if problems not found before system is
  operational
• What data would you collect? What trends would
  you expect to see evolve?

131
Requirements Metrics (Sample Solution)
Completeness and Volatility Analysis
Total Number of Requirements
1000
900
800
700
600
Quantity
500
400
300
200
100
0
1Q94
2Q94
3Q94
4Q94
1Q95
2Q95
3Q95
Calendar Quarter
CRR Looks Good! (Stable)
Combination of BOTH views indicates risk area -
requirements are NOT YET stable
132
Decide on Course of Action

• Purpose
• Ensure that project risks continue to be managed
  effectively
• Description
• Uses tracking data to determine how to proceed
  with project risks
• Close the risk
• Continue tracking and executing the current
  mitigation strategies
• Re-plan risk mitigation
• Invoke an alternate mitigation strategy (e.g.,
  use a contingency plan)

133
Execute Control Actions

• Purpose
• Implement both the decision made about a risk and
  mitigation strategy as well as to ensure that all
  decisions are appropriately documented for future
  reference and historical record maintenance
• Ensure approval and resources are allocated
• Description
• The process where control decisions are
  implemented

134
Risk Control Data Flow

• Decisions
• Replan
• Close
• Invoke contingency
• Continue tracking

• Status Reports
• Risks
• Mitigation plans

Control
evaluate
decide
Statement of Risk
execute
Context Impact Probability Timeframe Classificati
on Rank Plan Approach Status
Statement of Risk
Context Impact Probability Timeframe Classificati
on Rank Plan Approach Status
Project Data
135
Risk Information Sheet (After Tracking and
Control Phase)
136
Risk Control Phase Summary

• Control Decisions are based on current
  information as well as experience, and are
  required to respond to changing conditions in
  watched and mitigated risks
• Risk tracking and control should be integrated
  with standard project management practices risk
  management should be tailored for a project

137
Module 8

• Communicate Document

Control
Communicate Document
Track
Analyze
Plan
138
Overview

• What is communication?
• Relationship to other CRM Process functions
• Enablers to communication
• Barriers to communication
• Documentation of risks

139
Relationship To Other CRM Process Functions
140
Why Communicate Risks?

• Makes risks, plans, actions, concerns, exchanges,
  forecast, and progress known
• Ensures the visibility of risk information
• To enable all project members to participate in
  defining and managing risks
• Ensures understanding of risks and mitigation
  plans
• Establishes an effective, ongoing dialog between
  the manager and the project team
• Ensures appropriate attention is focused on
  issues and concerns

141
Why Document Risks?

• In order to track and collect risk information
  internally and externally
• Risks have a life cycle and eventually all risks
  go away
• Probability or impact goes to zero
• Risk becomes a problem
• Documenting the life cycle of risks
• Helps you learn what worked and didnt work
• Should help you avoid similar difficulties
• Provides the opportunity to help other projects
  learn from your experience (Input to Lessons
  Learned Database)

142
Risk Documentation Types

• Risk Management Plan (NPG 7120.5B, NPG 8000.4)
• Risk Implementation Plan
• Risk Information Sheets ()
• Prioritized Risk List (NPG 7120.5B)
• Risk Profile
• Risk Analysis Reports
• Risk Mitigation Status Reports
• Risk Databases ()
• Spreadsheet Risk Tracking Logs
• () Recommended for KSC risk management activities

143
Module 9

• Implementing Continuous Risk Management

Control
Identify
Communicate Document
Track
Analyze
Plan
144
Overview

• Frequently asked CRM implementation questions
• When do I start?
• Whos involved?
• What do I need?
• What should I choose?
• What actions should I take?
• Hints and Tips
• Things to watch out for

145
When do I Start CRM?
Opportunity
Actions
Pre-contract activity
Include risk management provisions in the
solicitation and statement of work.
Major project milestones (e.g., contract award)
Prepare for a major project decision point, and
the need to increase knowledge about risks for
improved strategic planning.
Major project review
Prepare for standard reviews, such as design
reviews, functional tests.
Best time to start is at the beginning!
146
Whos Involved? (1)
Role/Description
Responsibilities and Tasks
Sponsor (e.g., senior mgr., VP, division chief)

• Provide visible support and encouragement
• Reward effective management of risks

Project manager (responsible for ultimate success
of project)

• Provide resources and funding
• Reward effective management of risks
• Monitor progress

Champion (advocates new technology or process
within the project)

• Publicize and promote CRM
• Coordinate changes and improvements on
• the project

Change agents (plan and implement changes in
organizations and projects

• Assist with recommendations of plans
• Evaluate existing and new tools

147
Whos Involved? (2)
Role/Description
Responsibilities and Tasks
Facilitators (trained in meeting skills, conflict
resolution, tools, etc., - act individually or as
a team)

• Conduct training sessions
• Provide CRM expertise
• Provide consulting during evaluation of
• progress

• Encourage and support use of CRM within
• their teams
• Report risk information to project manager
• Evaluate progress within their teams

Technical managers (e.g., team or functional
leads, such as software/hardware manager, test
mgr., etc.)
Project personnel (e.g., software or hardware
engineers, testers, etc.)

• Add CRM activities to day-to-day
• operations
• Maintain open communication about risks

148
Core Risk Management Team
149
Core RM Team Key Responsibilities (1)

• Reviews current and previous risk issues to
  validate, classify and group risks at the project
  level
• Determines risk attributes (probability, impact,
  and timeframe) and reprioritizes risks as needed
• Determines if addition information is needed
  (trade-studies or research)
• Implements mitigation options (contingency plans,
  descope plans). Determines who is involved and
  assigns risk owner. Reassigns risks when
  necessary
• Adjusts action planning (mitigation plans,
  mitigation time frames, etc)

150
Core RM Team Key Responsibilities (2)

• Tracks, communicates and controls risks
• Makes control decision recommendations (analyze,
  decide, execute) to PM for all risks
• Makes recommendations to PM regarding CRM policy
  and the communication of risks
• Makes recommendations to PM regarding risk
  closure and/or acceptance
• Make recommendations to PM concerning allocations
  of resources to mitigate risks

151
You need . . . Organization Structure,
Internal/External Communication
Organization Structure
Internal Communication
External Communication
152
Risk Management Planning Documentation Options

• Risk Management Plan
• Guides project personnel through the tailored
  process, methods, and tools for managing their
  project risks
• Recommended and preferred risk management
  planning documentation approach for KSC projects
• Risk Management Implementation Plan
• Guides project personnel (in the pre-formulation
  phase) regarding how they intend to bring risk
  management into the projects infrastructure and
  processes
• Utilized primarily for extremely large
  programs/projects, where there is a need to
  establish sponsorship, resources, and
  infrastructure for the overall risk management
  approach
• Development of a Risk Management Implementation
  Plan is not envisioned for KSC projects (Refer to
  Tab L Module 11 for an example plan)

153
Risk Management Plan Elements(NPG 8000.4,
Section 2.7.4)

• Introduction/Overview of Risk Management process
• Project Organization and Responsibilities
• Risk Management activities and practices
  in detail
• Budget, resources, and milestones for risk
  management activities and risk mitigation
• Procedure for documenting risks
• Assumptions
• Technical Considerations
• Constraints
• De-Scope Options

154
Sample Risk Management Plan

• KSCs Advanced Technology Development Center
  (ATDC) Risk Management Plan can serve as an
  example for your project to use
• Take a few minutes to read the ATDC Risk
  Management Plan (After Module 11)

155
You need to . . . Utilize Established Project
Meetings

• Weekly Team Meetings
• Establish priority of teams risks
• Assign responsibility for new risks
• Review and approve mitigation strategies
• Monthly Project Meetings
• Presentation of the teams Top N risks (and
  mitigation strategies)
• Decisions of appropriate risk mitigation actions
• Determination regarding allocation of resources
  to implement risk mitigation strategies
• NOTE The above are examples your project team
  should utilize existing meetings, NOT separate
  Risk Management meetings

156
You need a. . . Defined Process and Data Flow
Example
Periodic meetings
Individual activities
Analyze Classify risks Evaluate risks
Identify risks
Periodic meetings
Individual activities
157
You must choose your . . . Methods and
Tools
Control
Identify
Baseline Identification and Analysis
Brainstorming
Periodic Risk Reporting
 Project Profile Questions
Project Metrics
Risk Information Sheet
Short TBQ
Track
Taxonomy-Based Questionnaire (TBQ)
TBQ Interviews
Voluntary Reporting

• Project Metrics
• Failure Modes Effect Analysis (FMEA)
• Fault Tree Analysis (FTA)

Analyze

• Project Metrics
• Statistical Process
• Control (SPC)

Affinity Grouping
Plan
Baseline Identification and Analysis
 Binary Attribute Evaluation
Comparison Risk Ranking
Multivoting
 Pareto Top N
 Potential Top N
Taxonomy Classification
Risk Information Sheet
 Top 5

• Tri-level Attribute Evalu