Americas Aerospace Quality Group Registration Management Committee

1
Americas Aerospace Quality GroupRegistration
Management Committee
Boeing Supplemental Oversight Assessment Results
January 22, 2008 Presented by Brian
Simons Boeing Oversight Representative The Boeing
Company
2
Topics for discussion

• Boeing supplemental oversight process
• Assessment Activities
• Strengths and Issues
• Corrective Action Expectations
• Assessment results
• Statistics
• Significance
• Corrective Action Responses
• Questions and comments

3
Boeing Supplemental Oversight Process

• Supplemental assessment activities
• Semi-annual CB performance review to identify 6
  month planned activity
• Risk-Based assessment for each CB using
• - Aggregate client (supplier) performance
  (product and QMS)
• - Previous Boeing oversight activity
• - Total quantity and change of client base
• Ad Hoc or unplanned (Throughout the year)
• - Primarily based on unsatisfactory supplier
  performance
• - BOR (Boeing Oversight Representative) Support
  Requests
• - Management requests
• - Field Representative questions and inquiries
• Approximately 80 of Boeings supplemental
  oversight is Ad Hoc

4
Boeing Supplemental Oversight Process Jan
June 2008 - Americas Sector - Oversight
Opportunities
Y Axis Influence
X Axis Performance
5
Boeing Supplemental Oversight Process

• Types of supplemental oversight activity
• OASIS database and certificate information
  validation
• Audit report review (Audit summary, AS9101
  checklist, non-conformances, corrective action,
  etc.)
• Client validation (on-site, including audit
  report review)
• - Required for risk CBs identified on RAT
• CB auditor witness audit
• CB/ANAB office audit

6
Boeing Supplemental Oversight Process

• Strengths and Issues (SI) document
• Issue (CA required) or Comment
• Issues documented in a 3 part format
• Requirement Reference clause number and
  verbatim statement of requirement (e.g. AS9104
  clause 8.4 The AQMS audits shall be documented on
  the Aerospace Assessment Report.).
• Issue (Nonconformance) Restatement of
  requirement in the negative (e.g. The AQMS audits
  are not documented on the .)
• Evidence Documentation of evidence collected,
  proceeded by For example (e.g. For example, no
  evidence of completed AS9101 checklist for
  surveillance assessment 123, dated 1/22/08)
• The BOR team will begin providing ANAB with a
  courtesy copy of all SIs issued to CBs beginning
  2008

7
Boeing Supplemental Oversight Process

• Sample SI - Issue 1
• Requirements
• AS9104 clause 8.3 Nonconformity No certificates
  or approvals to AQMS Standards or any combination
  of AQMS Standards with ISO 9001 shall be issued
  unless all major and minor nonconformities have
  been satisfactorily corrected with the root cause
  analysis and corrective action verified by the
  CRB.
• IAF Guidance on the Application of ISO/IEC Guide
  62 G.3.8.5. The implementation of the corrective
  action should not be deemed to have been
  completed until its effectiveness has been
  demonstrated and the necessary changes made in
  the procedures, documentation and records.
• Issue
• The CRB does not verify the satisfactory
  implementation, completion and effectiveness of
  corrective action for nonconformities, including
  demonstration that the necessary changes were
  made in the procedures, documentation and
  records, as necessary.
• Evidence
• For example, The client corrective action for
  NCRs 2 and 3 from the recertification audit
  were not completed prior to CB X closing the
  NCRs. NCR 2 client corrective action states that
  inspector training will be completed by February
  15, 2007. The training record provided by the
  client for this action is dated 4/4/2007. NCR 2
  was closed on 2/5/07. NCR 3 client corrective
  action states that before February 10, 2007 the
  client will implement a revision to the
  procedures manual addressing AS9100 clause
  7.5.1.3. As of November 1, 2007, there is no
  evidence of a revision to the client's Quality
  Procedure Manual for clause 7.5.1.3 (Procedure
  does not exist) or the Quality Assurance Manual
  (current revision December 2003). NCR 3 was
  closed on 2/5/07.

8
Boeing Supplemental Oversight Process

• SI Corrective Action expectations (5 part
  format)
• Immediate Correction
• - Address and correct the evidence
• - Identify the Direct Cause
• - Notify affected personnel
• Root Cause Statement
• - Identify fundamental process deficiency
• - Personnel error or mistake is unacceptable
• Root Cause Corrective Action (CA) Plan
• - Address the Root Cause
• - Identify what, who and when
• - Include reference to revised QMS policy,
  process, etc. (title/number)

9
Boeing Supplemental Oversight Process

• SI Corrective Action expectations (5 part
  format)
• (Continued)
• Verification of Corrective Action Plan
• - Verify RCCA actions have been completed per
  commitments
• - QMS process updated, training completed,
  clients assessed, etc.
• Follow-up audit
• - Determine of CA plan is effective in
  preventing recurrence
• - Conducted 6-9 months following verification

10
Boeing Oversight Assessment Results

• BOR oversight activity from January 1, 2006
  through December 31, 2007
• 19 out of 34 domestic Certification Bodies
  Assessed (CBs associated with Boeing suppliers)
• CBs assessed based on RAT and Ad Hoc requests
• 69 out of 79 oversight activities resulted in at
  least 1 issue
• (87 rejection rate)
• 32 - OASIS database and Certificate validations
• 26 - Client Validations (includes audit report
  review)
• 9 - Audit Report Reviews (Desktop review)
• 1 - Office Audit
• 1 - Witness Audit
• 143 total issues identified (Nonconformance to
  industry requirement)

11
Boeing Oversight Assessment Results
12
Boeing Oversight Assessment Results

• Supplemental oversight from January 1, 2006
  through December 31, 2007
• Inadequate QMS assessment (31)
• OASIS update gt 30 days (21)
• Audit report deficiency (16)
• Multi-site issue (13)
• Assessment days below minimum requirement (11)
• CB contract deficiency (7)
• Certificate deficiency/error (7)
• Certificate issued with incomplete client
  corrective action (7)
• NC soft grading (6)
• OASIS errors (5)

13
Boeing Oversight Assessment Results

• Supplemental oversight from January 1, 2006
  through December 31, 2007
• Certificate validity
• Inadequate QMS assessment (31)
• Audit report deficiency (16)
• Multi-site issue (13)
• Assessment days below minimum requirement (11)
• Certificate issued with incomplete client
  corrective action (7)
• NC soft grading (6)
• CB Administration
• OASIS update gt 30 days (21)
• CB contract deficiency (7)
• Certificate deficiency/error (7)
• OASIS errors (5)

14
Boeing Oversight Assessment Results

• Inadequate Quality Management System Assessment
  (31)
• Ineffective QMS management processes (Client
  validation)
• Management Review, Quality Objectives, Internal
  Audit, Corrective Action, Analysis of Data,
  Continual Improvement, etc.
• Ineffective product control processes (Client
  validation)
• Configuration Management, Production
  Documentation, Identification, Accountability,
  Control of Nonconforming Product, etc.
• Completed audit reports lack sufficient evidence
  to justify satisfactory result
• - Prompts Boeing oversight investigation and
  results comparison
• Unsatisfactory Boeing supplier performance
• Compare Boeing 2nd party QMS audit results vs. CB
  audit results
• Prompts Boeing oversight investigation

15
Boeing Oversight Assessment Results

• Certification Body Corrective Action Response
• Corrective action response due within 30 days
  (typical)
• Initial submittal rejection rate approx. 90-100
• - 5 part format not followed
• - Evidence cited in SI not addressed and
  corrected in Immediate Action
• Root Cause identifies human error or mistake
• Root Cause is an explanation or rationalization
  of the condition
• Root Cause Corrective Action does not address
  Root Cause
• Root Cause Corrective Action does not address
  long term process improvement
• CA Verification and Follow-up are confused and
  mixed together
• BOR policy is to elevate unsatisfactory SI CA to
  ANAB for resolution
• - 2 SIs were elevated in 2007 for ANAB
  investigation and action

16
Thats all folks!

• Questions or Comments?