Programmatic Audit and Review PA

1
(No Transcript)
2
Module II - Knowledge

• Make Decision to Audit
• Conduct Program Discovery
• Understand Baseline SMA Requirements

3
PAR Knowledge Steps
Module II
In this module, we will discuss the PAR Steps
that Initiate the PAR Process Develop Knowledge
of Program SMA Requirements
Remember this chart from the Introduction which
shows the Principal Steps of the PAR Process.
4
Details of the PAR Process
Module II

• This module provides a detailed discussion of the
  principal knowledge steps. The steps will be
  depicted in a simplified flowchart as shown
  below.

Decide
Discovery
Flow-down
Capability
Compliance
Understand Requirements
Feedback
Operational Readiness
Additionally, the step being discussed will be
highlighted, e.g., Flow-down.
5
P1 - Decide Make Decision to Audit
Module II
The Decide Process Step is the kick-off of the
PAR Process select what programs/ projects to
audit/review/assess, determine who should lead
the audit and define the initial scope of the
PAR.

• Program/Project Selection
• Select program/project for PAR
• By Risk-based Selection Logic
• By Request
• By Direction
• Determine if PAR will be led by Headquarters or
  a Center
• PAR Planning Workgroup
• PAR Initiation
• Designate PAR Audit/Review Lead
• Establish PAR Team
• Initiate contact with the program or project.
• Define preliminary scope and schedule.
• Draft PAR Initiation Letter for signature and
  distribution.

6
Program Selection
Module II
HQ-led List
Chief, SMA Approval
HQ-led Plan
Analysis
PAR Planning Workgroup
Risk/ Rule- Based Filter
MSD Filter
Preliminary Selection and Scoping
RAD Filter
Program/ Project Information Sources
Center-led List
Chief, SMA Info
Integrated Schedule
START
Center-led Plans
Center SMA Planning

• The above flowchart depicts the PAR Program
  Selection for both Headquarters-led (upper flow)
  and Center-led (lower flow) PARs.
• The HQ-led PAR process will be discussed in
  detail in the following slides.
• NASA Centers have their own internal processes
  for selection that may be
• similar to the NASA HQ process.
• The PAR selection process for both HQ and
  Centers comes together at the PAR
• Planning Work Group

7
Program Selection
Module II
HQ-led List
Chief, SMA Approval
HQ-led Plan
Analysis
PAR Planning Workgroup
Preliminary Selection and Scoping
Risk/ Rule- Based Filter
MSD Filter
RAD Filter
Program/ Project Information Sources
Center-led List
Chief, SMA Info
Integrated Schedule
START
Center-led Plans
Center SMA Planning

• The PAR Process begins with the collection and
  analysis of program or project information from
  Agency sources.
• The step continues with the development of the
  initial list of candidates for HQ-led PARs using
  broad-based filters, reviews of previous audit
  activities, and the OSMA insight into
  programs/projects. The various filters are listed
  on the next slide.

8
Program Selection
Module II
HQ-led List
Chief, SMA Approval
HQ-led Plan
Analysis
PAR Planning Workgroup
Preliminary Selection and Scoping
Risk/ Rule- Based Filter
MSD Filter
RAD Filter
Program/ Project Information Sources
Center-led List
Chief, SMA Info
Integrated Schedule
START
Center-led Plans
Center SMA Planning

• Risk/Rule-Based Filter
• Human Safety
• Nuclear Safety
• Multi-Center
• Multi-National
• High Cost
• Center
• Mission Directorate
• Mishaps/Audits/Reviews
• Lifecycle phase

• MSD Filter
• Program/Project Insight
• Contacts with program/project SMA management
• Contacts with Center SMA management
• Contacts with Mission Directorate management

• RAD Filter
• Previous RAD audit activities
• Other independent audit/review activities

9
Program Selection
Module II
HQ-led List
Chief, SMA Approval
HQ-led Plan
Analysis
PAR Planning Workgroup
Preliminary Selection and Scoping
Risk/ Rule- Based Filter
MSD Filter
RAD Filter
Program/ Project Information Sources
Center-led List
Chief, SMA Info
Integrated Schedule
START
Center-led Plans
Center SMA Planning

• Using a standard template, each center and HQ
  develop and revise, as appropriate, their PAR
  activities.
• Selection and schedule primarily based on the
  PAR risk-informed analysis/ranking.
• Additional candidates may be identified by
  request from an entity within NASA (e.g., NASA
  Administrator, Mission Directorate AA, Chief,
  SMA, etc.) or by direction from a higher
  authority external to NASA (e.g., GAO, Congress,
  etc.).

10
Program Selection
Module II
HQ-led List
Chief, SMA Approval
HQ-led Plan
Analysis
PAR Planning Workgroup
Preliminary Selection and Scoping
Risk/ Rule- Based Filter
MSD Filter
RAD Filter
Program/ Project Information Sources
Center-led List
Chief, SMA Info
Integrated Schedule
START
Center-led Plans
Center SMA Planning

• The PAR Planning Workgroup (PWG) is chartered to
  plan, coordinate, and fully integrate Agency-wide
  PAR activities. The current status of the PAR
  activities is presented at the annual PWG
  meeting.
• The RAD PAR Manager formally develops/presents a
  list of potential HQ-led PAR candidates.
• Each PWG Center Representative identifies current
  or planned PARs for their respective Center.
  These lists contain justification for performing
  the PAR, preliminary scoping, and the name of
  the proposed PAR Lead.
• HQ and Center PAR candidates are discussed as to
  their merit for inclusion/deletion/ combination
  into the proposed integrated list.

11
Program Selection
Module II
HQ-led List
Chief, SMA Approval
HQ-led Plan
Analysis
PAR Planning Workgroup
Preliminary Selection and Scoping
Risk/ Rule- Based Filter
MSD Filter
RAD Filter
Program/ Project Information Sources
Center-led List
Chief, SMA Info
Integrated Schedule
START
Center-led Plans
Center SMA Planning

• Candidate List Presented to Chief, SMA
• The proposed lists of programs and projects
  developed by the PWG are presented to the Chief,
  SMA for approval (Headquarters-led PAR) and
  information (Center-led PAR).
• A rough order of magnitude (ROM) estimate of the
  resources (and their source) to perform the PAR
  is also provided.
• Once approved, the integrated schedule is made
  available via the web-based PBMA-KMS for agency
  wide access.

12
Program Selection
Module II
HQ-led List
Chief, SMA Approval
HQ-led Plan
Analysis
PAR Planning Workgroup
Preliminary Selection and Scoping
Risk/ Rule- Based Filter
MSD Filter
RAD Filter
Program/ Project Information Sources
Center-led List
Chief, SMA Info
Integrated Schedule
START
Center-led Plans
Center SMA Planning

• By coordinating and maintaining an annual,
  integrated risk-informed PAR schedule
• Programs and centers can improve resource
  planning
• Overlapping and redundant efforts can be
  reduced
• PAR impact on program/project schedules is
  minimized

13
PAR Initiation
Module II
Approval
Select PAR Audit/Review Lead
Select PAR Team
Propose PAR Scope and Schedule
Draft PAR Initiation Letter
Distribute

• PAR Manager (for Headquarters-led) or the Center
  SMA Director (for Center-led) selects the PAR
  Lead, who will have the overall responsibility
  for the successful execution of the PAR.
• PAR Lead recruits appropriate personnel to serve
  on the PAR Team to perform specific PAR
  activities, as scoped. Team may consist of both
  civil servants and/or support service
  contractors.

14
PAR Initiation
Module II
Approval
Select PAR Audit/Review Lead
Select PAR Team
Propose PAR Scope and Schedule
Draft PAR Initiation Letter
Distribute

• The PAR Team meets and drafts a plan (scope) for
  conducting the PAR. Scoping contains the
  following elements
• 1st Order Which step or steps will be performed
  (e.g., Program Discovery, Understand
  Requirements, Requirements Flow-down
  Verification, Process Capability Assessment,
  On-site Audit)
• 2nd Order Organizational level (e.g., program,
  project, prime, first-tier subs, second-tier
  subs)
• 3rd Order Requirement/Process Scope (e.g.,
  sample vs. total requirements set, selected
  processes)
• 4th Order Audit Depth
• Simple analysis yes/no checklist format
• Complex analysis - quality assessment of plan
• The following slides provide a notional scoping
  matrix that can be used to formally document the
  scoping process and examples of simple vs.
  complex.

15
Notional Scoping Matrix
Module II
Resource and time constraints require careful
audit/review scope definition to maximize return
on investment.
16
Simple vs. Complex Analysis
Module II
17
PAR Initiation
Module II
Approval
Select PAR Audit/Review Lead
Select PAR Team
Propose PAR Scope and Schedule
Draft PAR Initiation Letter
Distribute

• The PAR Initiation Letter is the formal
  notification to the program or project that the
  PAR will be conducted. It contains the PAR
  plan, including the preliminary scope and
  schedule of the PAR. The letter also identifies
  the PAR Lead.
• Chief, SMA or Center SMA Director approves scope
  and Initiation Letter.
• The PAR Initiation Letter is then distributed to
  appropriate Program, Project, and SMA personnel.

After the letter is distributed, scoping is
finalized through detailed discussion with the
program or project.
18
P2 - Conduct Program Discovery
Module II
The Discovery Process Step has a two-fold
purpose Educate the PAR Team and instill
confidence in the Project/Program of the PAR
team knowledge and the credibility of results.

• Characterize Program
• Provide Context for PAR
• Documentation Review
• Interviews
• Technical Interchange Meetings
• Briefly summarize program or project
  (program/projects mission, objectives, goals,
  background, and SMA processes) to ensure PAR
  Team is prepared
• Describe SMA Roles, Responsibilities and
  Relationships
• Review existing Program/Project developed SMA
  process map and matrix, if available
• Assist in the development of Map and Matrix

19
Characterize Program
Module II
Example using X-37
Data Sources RFP Interviews Proposal Public
Information Program Office Etc.

• PAR Team gathers information from
• Center SMA, Program, Project, and
• Contractor via document review and
• TIMS
• Amount and level of detail of
• information is directly dependent on
• the scope of the PAR and the
• criticality and complexity of program/
• project under review.

Information To Provide Overall Program/Project
Context
Program Selection
By conducting this step, the PAR team gains
knowledge and the program/project gains
confidence in the capability of the PAR team and
credibility of the results of the process.
20
Describe SMA Roles, Responsibilities and
Relationships
Module II

• Ideally, program/projects have already developed
  SMA process maps and matrices as required by NPR
  7120.5D and NPR 8705.6 and have provided this key
  source of information as part of the
  Characterization step. However, if the
  map/matrix has not been developed, the PAR team
  may require and assist in their development.
• SMA Process Analysis results in an SMA process
  map and matrix that provide a concise graphical
  and narrative representation of the safety and
  mission assurance roles, responsibilities, and
  relationships for assurance agents supporting
  safety and risk management within a given program
  or project.
• The creation and maintenance of the SMA Process
  Map and Matrix is the responsibility of the
  program or project. Because this is a new tool,
  OSMA RAD will support program or project SMA in
  creating the document.
• The following slides show an example of an SMA
  process map and matrix. For more detailed
  information on the SMA Process Analysis please
  refer to the AP Map Survival Kit found on PBMA at
  
• http//pbma.nasa.gov/apmaps_main_cid_568

21
Describe SMA Roles, Responsibilities and
Relationships
Module II
SMA Process Map

• The program or project develops a baseline SMA
  Process Map using SMA organizational charts,
  program organizational charts, and other
  information gained through technical interchange
  meetings and interviews with Center and program
  SMA organizations, coupled with any other
  relevant information discovered during the
  Characterize Program step.

22
Describe SMA Roles, Responsibilities and
Relationships
Module II
This is a representative page of an SMA Process
Matrix
23
Describe SMA Roles, Responsibilities and
Relationships
Module II
Ideally, the SMA Process Map and Matrix should
be developed early in the program/project and
used as a management tool The SMA
Process Map and Matrix should be updated
throughout the program/project lifecycle
The Map and Matrix are required inputs to the
Safety and Mission Success Review (SMSR)
Identify Accountable Organizations
Notional Mission Assurance Process Map
Identify Organization Roles and Points of Contact
Notional Mission Assurance Process Matrix
24
P3 - Understand Baseline SMA
Requirements
Module II
A meaningful review, assessment, or audit of a
program or project requires a thorough
understanding of the BRS.

• Analyze Baseline Requirements Set (BRS)
• Principal purpose of this step is to simply
  document the existing BRS for the project/program
  to be used throughout the PAR process.
• As appropriate, perform a requirements gap
  analysis of the programs BRS, all applicable
  Agency SMA requirements, and accepted best
  practices. Document gaps and assess/document
  residual risks

25
Analyze Baseline Requirements
Set (BRS)
Module II
The PAR Team works with program/project SMA and
the Office of the Chief Engineer (OCE), OSMA
Mission Support Division and Safety and Assurance
Requirements Division (or the Center equivalent
organization for a Center-led PAR) to develop a
complete list of the Agency-level SMA
requirements documents that are imposed on the
program or project (the BRS) and SMA requirements
that should be on the program or project. The
PAR Team also uses Agency knowledge management
resources to assemble a list of best practices
and lessons learned that are relevant to the
program or project.
26
Analyze Baseline Requirements
Set (BRS)
Module II
If deemed appropriate, a Gap Analysis can be
performed as a series of traceability
assessments. The initial traceability
assessment compares the program-applicable Agency
SMA requirements set to the contractual
requirements set. This analysis identifies those
program-specific Agency requirements not
traceable to the program/project contractual
requirements (e.g., those requirements
negotiated-out or subject to subsequent Agency
SMA requirements revisions, additions, or
deletions). Additionally, the PAR Team can
perform a requirements gap analysis between the
existing set of program contractual SMA
requirements, any program-applicable Agency SMA
requirements, and a set of accepted engineering
best practices.
27
Analyze Baseline Requirements
Set (BRS)
Module II
The untraceable requirements from the Gap
Analysis represent a delta requirements set.
Requirements in the delta requirements set are
then prioritized by their potential increased
risk to safety and mission success. The output
of the traceability assessments is a SMA
requirements risk assessment that is used to
inform the PAR Team as the PAR Process proceeds
and to provide the Program, Project, Center,
Mission Directorate, Chief, SMA and the Chief
Engineer with feedback as to the
program/projects SMA requirements status
28
Analyze Baseline Requirements
Set (BRS)
Module II

• In summary, the gap analysis identifies missing,
  redundant, or new/revised requirements resulting
  in
• Feedback to MSD (process step P7) or equivalent
  Center organization for inclusion of missing
• requirements based on negotiation with
  Program Management
• Feedback to SARD (process step P7) suggesting
  improvements to be included in Agency
• SMA Requirements Set
• Identification of any residual risks related to
  gaps