Programmatic Audit and Review PA

1
(No Transcript)
2
Welcome

• Welcome to the Programmatic Audit and Review
  (PAR) Process training module. Module I begins
  with a basic introduction to the elements of the
  Office of Safety and Mission Assurance (OSMA)
  Review and Assessment Division (RAD) and provides
  a detailed description of the Programmatic Audit
  Review (PAR) Process.
• The PAR Process directly supports the OSMA/RAD
  charter to ensure lifecycle safety and mission
  success through compliance verification of
  Agency SMA requirements thus providing a robust
  foundation for the Safety and Mission Assurance
  (SMA) operational readiness decisions.

3
Training Agenda

• Module I - Introduction and PAR Overview
• Module II- Knowledge
• Make Decision to Audit
• Conduct Program Discovery
• Understand Baseline SMA Requirements
• Module III - Visibility
• Verify Requirements Flow-down
• Verify Process Capability
• Verify Requirements Compliance
• Provide Feedback
• Module IV Understanding
• Support Operational Readiness Decision
• Module V Summary

4
Module I - Introduction
Overview

• Why Conduct a PAR?
• What is a PAR?
• How Do We Implement the PAR Process?

5
Why Conduct a PAR?
Module I
Prepare a detailed plan for defining,
establishing, transitioning, and implementing an
independent Technical Engineering Authority,
independent safety program, and a reorganized
Space Shuttle Integration Office

Recommendation R9.1-1 of the CAIB Report dated
August 2003

• The Programmatic Audit and Review (PAR) Process
  was initiated in response to the Columbia
  Accident Investigation Board (CAIB) Findings and
  Recommendations to provide a robust, structured
  and formalized independent Safety and Mission
  Assurance Compliance Verification function in the
  Agency. It represents, in part, the Agency
  response to CAIB recommendation R9.1-1 and R7.5-2
  on the need to increase independent assessment
  and compliance verification activities, and is in
  accordance with commitments by the Administrator
  to initiate Space Shuttle Program independent
  review and assessment activity prior to Return to
  Flight (RTF).

NASA Headquarters Office of Safety and Mission
Assurance should have a direct line authority
over the entire Space Shuttle Program safety
organization and should be independently
resourced.
Recommendation R7.5-2 of the CAIB
Report dated August 2003
This PAR function was established under a new
Review and Assessment Division of the Office of
Safety and Mission Assurance in 2003 and was
chartered via the normal NPR processes.
6
Office of Safety and Mission
Assurance
Module I
The Office of Safety and Mission Assurance
(OSMA), as well as Center Safety Mission
Assurance (SMA) organizations, play a key role in
the PAR Process. The chart below depicts the
PAR-related responsibilities of OSMA. NASA
Centers may have similar relationships. Although
all divisions of OSMA have a role, RAD has the
primary OSMA compliance verification
responsibility.
7
The RAD Compliance Verification
Landscape
Module I
8
The RAD Compliance Verification
Landscape
Module I
Compliance Verification of Agency SMA Requirement
Set
Agency SMA Review Assessment
Functional Audits
Agency Requirements
Institutional / Facility / Operational (IFO)
Safety Audit
Institutional Programmatic Support (IPS) Audit
Center Requirements
Program Requirements
IFO Safety Audits are conducted on site at the
NASA Centers to verify compliance with Federal,
State, local, and Agency SMA and technical
directives, policies, and requirements contained
in applicable documents.
These requirement areas within the IFO Baseline
Requirements Set (BRS) include, but are not
limited to Public/Worker Safety, Facility System
Safety, Mishap Investigation, GIDEP, Fire
Protection, Equipment Safety, Explosives Safety,
Range Safety, and Electrical Safety.
Programmatic Audit Review (PAR)
Contracts
Contractor Requirements
9
The RAD Compliance Verification
Landscape
Module I
Compliance Verification of Agency SMA Requirement
Set
Agency SMA Review Assessment
Functional Audits
Agency Requirements
Institutional / Facility / Operational (IFO)
Safety Audit
Institutional Programmatic Support (IPS) Audit
Center Requirements
IPS Audits are conducted onsite at the NASA
Centers to verify that Institution/Center
policies, procedures and processes are in place
to define, impose, and implement appropriate
Agency SMA requirements for their respective
programs and projects.
Program Requirements

• The purpose of the IPS is twofold
• Provide management with an independent,
  objective, and constructive evaluation of the
  compliance of the institution with the applicable
  IPS BRS.
• Assess the effectiveness of implementation of
  SMA process and technical directives, policies,
  and requirements in achieving desired outcomes
  relevant to the areas being audited.

Programmatic Audit Review (PAR)
Contracts
Contractor Requirements
10
The RAD Compliance Verification
Landscape
Module I
Compliance Verification of Agency SMA Requirement
Set
Agency SMA Review Assessment
Functional Audits
Agency Requirements
Institutional / Facility / Operational (IFO)
Safety Audit
Institutional Programmatic Support (IPS) Audit
Center Requirements
Program Requirements
Programmatic Audit Review (PAR)
Contracts
The PAR process provides independent compliance
verification of Agency SMA process, technical,
and, as appropriate, engineering performance
specification (EPS) requirements within the
applicable program/project BRS.
Contractor Requirements
PARs include analyses, reviews, and assessments
in addition to the on site compliance
verification audits at center, prime contractor,
and subcontractor facilities as appropriate
11
The RAD Compliance Verification
Landscape
Module I
12
Authorizing Documents
Module I
There are several Agency level documents that
authorize compliance verification and the PAR
Process
NASA Policy Directive NPD 1000.0, Section 3.2
states that NASA employs a system of checks and
balances for effective internal control and to
ensure the successful achievement of missions,
assigning proper levels of influence and action
to different organizations. The Safety
Mission Assurance organization maintains
responsibility for verification of programmatic
compliance through strategies, policies, and
standards.
NASA Policy Directive NPD 1000.3C, Section
4.6.2.2 (d) states that the Chief, Safety and
Mission Assurance is responsible for (d)
Verifying the effectiveness of SMA requirements,
activities, and processes.
NPR 7120.5D, paragraph 2.5.4 states that NASA HQ
SMA also has a Programmatic Audit and Review
(PAR) process described in NPR 8705.6, Safety
and Mission Assurance Audits, Reviews, and
Assessments. That process provides independent
compliance verification for the applicable NASA
SMA process and technical requirements within the
program/project safety and mission assurance
plan, the program baseline requirements set, and
appropriate contract documentation.
Program/Project Managers directly support the
PAR process (either Headquarters-led or
Center-led) by providing logistics and resource
support required for the successful execution of
and response to PAR process activities. They
also coordinate with Center SMA and Center
procurement officials to ensure that contracts
provide for adequate contractor support for all
PAR activities, and they direct and authorize
program/project contractors to support PAR
process activities.
NPD 8700.1C, Section 1.e states that it is NASA
policy to verify and validate life-cycle
implementation of SMA, RM, and mission- success
requirements through ongoing surveillance of
program, project, and contractor processes.
13
Governing Documents
Module I
There are two Documents Governing the PAR Process
Headquarters Office Work Instruction HOWI
8700-GE037
NASAwide NASA Procedural Requirement NPR 8705.6
For complete copies of these documents, go to
the NASA On-line Directive Information System
(NODIS) Library at http//nodis.hq.nasa.gov/
14
What is a PAR?
Module I

• The Programmatic Audit and Review (PAR)
  Process is designed to provide independent
  verification of NASA Center, Program, or Project
  compliance with Agency Safety and Mission
  Assurance (SMA) requirements as reflected in the
  applicable programmatic SMA Baseline Requirement
  Set (BRS). The PAR Process
• Is an Agency-wide process, elements of which are
  owned by a variety of HQ and Center
  organizations.
• Helps preserve and protect value created by the
  program or project.
• Is a presence throughout the program/project
  lifecycle, from initial planning stages to
  end-of-life.
• Supports Operational Decisions with
  Audit/Review/Assessment Reports and the
  identification and qualitative assessment of SMA
  requirements risk.

A set of SMA requirements jointly negotiated
among the program/project, engineering community,
SMA community, and, as appropriate, institutional
organizations imposed on a program/project.
Typically, the BRS is a subset of Agency SMA
process, technical, and engineering performance
specification requirements uniquely applicable to
a given NASA program, project, facility, or
operation.
15
Further Elaboration on the PAR Process
Module I

• What PAR is
• An audit, assessment, or review of a program or
  project activity conducted by a PAR team of
  Subject Matter Experts (SMEs) independent of the
  activity
• A focused look, in both time-line and scope, to
  provide specific knowledge, visibility, and
  understanding into a program or project
• PARs can look at any given level of the program
  or project organizational hierarchy at any phase
• A process that results in specific findings,
  corrective actions, and a final report and
  out-briefing to the Chief, Safety and Mission
  Assurance and the Office of Chief Engineer
  (HQ-led PAR) or the Center SMA Director
  (Center-led PAR).
• What PAR is NOT
• Continuous, day-to-day, matrixed SMA support to a
  program or project

16
White Hat vs. Black Hat
Module I

• PAR is designed to be flexible in a White
  Hat/Black Hat application of audits, assessments
  and reviews. PAR results vary by PAR
  categories and include both recommendations and
  findings. These terms are formalized in the
  following slides.

White Hat Provides advice/support/mentoring to
the person/organization observed. White hat
assessor looks at goodness of a process.
Black Hat Assures that policies, procedures,
and practices are in accordance with requirements
through the implementation of rigorous audit
processes.
17
PAR Categories
Module I
Assessment An evaluation and analysis of
procedures, processes, and practices necessary to
effectively implement requirements. The
assessment findings are provided for information
to the organization being assessed.
Review An examination and analysis of
documentation to evaluate the feasibility,
appropriateness, relevance, and/or effectiveness
of documentation contents. It is a necessary
precursor to an audit or assessment. The review
findings are provided for information to the
organization being assessed.
Audit An onsite in-process verification of
compliance with policies, procedures, processes,
and requirements. Audits have the responsibility
for citing non-compliances, as well as
observations, and require a follow-up corrective
action process, including a corrective action
plan and corrective action status report(s).
18
PAR Findings
Module I
Critical Non-compliance (CNC) A failure to
comply with a program/project SMA plan
requirement, and/or a contract SMA requirement
that could lead to 1) loss of life or serious
injury, and/or 2) loss of a high value asset,
and/or 3) impairment of mission success. A CNC
will require immediate remedial action.
Non-compliance (NC) A failure to comply with a
program/ project SMA plan requirement, and/or a
contractually imposed SMA requirement.
Observation (OBS) A finding which is
technically not a non-compliance but has the
potential to adversely affect safety and/or
mission success. Observations may also include
identification of potential best practices.
19
How Do We Implement the PAR Process?
Module I

• This section of the training generally describes
  how we implement the PAR Process. It introduces
  the Principal Steps and how they fit into the
  traditional Program/Project Lifecycle

20
Top-Level PAR
Goals and Process
Module I
These are the Principal Steps of the PAR
Process. A general discussion follows. Modules
II, III, and IV provide a detailed discussion of
each of step.
21
Top-Level PAR
Goals and Process
Module I

• Make risk-based selection determined by
• direction, request, or formal selection logic
• Finalize schedule through PAR Planning
• Workgroup
• Notify program or project of selection
• Develop initial scope
• Establish the PAR Team

22
Top-Level PAR
Goals and Process
Module I

• Working with Program/Project POC, compile
  documentation and information to learn about the
  program
• PAR team briefly summarizes program or project
• SMA processes
• PAR reviews Project/Program developed SMA
• Maps and Matrices which identify SMA roles,
• responsibilities, and relationships

23
Top-Level PAR
Goals and Process
Module I

• Obtain the program-specific SMA
• baseline requirements set (BRS)
• for each program or project
• Understand the risk associated
• with any SMA requirements gaps

24
Top-Level PAR
Goals and Process
Module I

• Review the formal documented flow-down of the BRS
  through the organizational chain (program to
  project to prime contractor to subcontractor)
• Propose recommendations, as appropriate.

25
Top-Level PAR
Goals and Process
Module I

• Review key SMA Processes
• Assess each implementing organization
• (any level in the organizational chain)
• to ensure there are sufficient SMA
• resources (i.e., staffing, skill mix, tools,
• funding, etc.)
• Propose recommendations, as appropriate.

26
Top-Level PAR
Goals and Process
Module I

• Conduct a rigorous, in-process, on-site
• compliance verification audit to verify
• implementation of baseline requirements
• Document audit findings which consist of
• non-compliances and observations
• Track and verify corrective action

27
Top-Level PAR
Goals and Process
Module I
Inform all SMA process stakeholders of the PAR
results and any suggested revisions to
Agency-wide or program-specific requirements
28
Top-Level PAR
Goals and Process
Module I
Summarize PAR findings and SMA requirement
risks for the Chief, Safety and Mission
Assurance, the Chief Engineer or appropriate
Center-level decision maker
29
PAR A
Lifecycle Approach
Module I

• The PAR Process is designed to be applied during
  any phase or before/between any major decision
  points of a program or project lifecycle, e.g.
  Program Milestone Reviews such as SRR, PDR, CRR,
  FRR and LRR. Emphasis typically evolves from
  concentration on flow-down and capability
  verification early in a program or project to
  rigorous compliance verification in later
  lifecycle phases.
• Because no two programs/projects are identical,
  the PAR Process adapts to the unique complexity
  of each program or project, taking full advantage
  of information gained in earlier phases, as well
  as other internal and external audits and reviews.

30
PBMA-KMS Provides Key Support to the PAR
Process
Module I

• The Process Based Mission Assurance Knowledge
  Management System (PBMA-KMS) provides quick and
  easy access to critical Safety and Mission Data
  to support program managers and SMA
  professionals.
• PBMA-KMS functionality is being actively applied
  to manage the PAR process
• Enhanced Security Work Groups are used to manage
  and support PAR activities
• Library functionality used to distribute team
  data, requirements documents, briefing materials,
  and audit reports to team members
• Audit/Review Findings securely stored via Secure
  But Unclassified (SBU) capability
• SecureMeeting is used to share real-time
  sensitive information and conduct meetings
  remotely via the Internet.
• The Knowledge Registry is used to identify
  individuals with the best SMA / technical
  expertise to support PAR audits/reviews.

• To access PBMA go to

http//pbma.nasa.gov