Software Quality Assurance

1
Revision to the Evaluation of Sellers QA
Program Subject Area (SA) Now SUPPLIER
EVALUATION
Quality Rep. Workshop (11/30/2005)

• Background / Discussion
• Improvements/Requirements
• Demo SES ISIS
• Conclusion

2
Discussion/Background

• Opportunity for Improvement identified during
  internal assessment of ERDs procurement quality
  practices
• Pre-Award Evaluation of Suppliers QA Program
  not performed for risk-significant (A1/A2)
  procurements as required by current SA.
• SA does not clearly define how to evaluate
  suppliers without QA Programs (e.g. service
  providers)
• Further review of risk-significant (A1/A2)
  procurements from new suppliers found evaluations
  either not being performed or not documented (no
  objective evidence)

3
Team Members

• Joe Labas (SME), QMO/PPM
• Steve Stein, QMO
• Dave Passarello, CA-D
• Mike Buckley, NSLS
• Stasia Scocca, QMO
• John Selva, EWMS
• Kevin Corbett, ERD
• Reviewed by
• Ann Emrick, Life Sciences
• Diane Cabelli, Chemistry
• Pat Carr, EENS

4
Drivers External Requirements

• DOE Quality Order (DOE O 414.1) and
• Rule (10CFR830.122)
• Criterion 7 Procurement
• Procure items services that meet established
  requirements
• Evaluate and select prospective suppliers on the
  basis of specified criteria
• ensure approved suppliers continue to provide
  acceptable items and services.
• Criterion 8 - Inspection Acceptance Testing
• Inspect and testusing established acceptance
  performance criteria.

5
Current Drivers KEY Assessments FY06

• OA - 40 ISM Program Assessment - FY06
• Similar to 2000 verification Assessment
• Large DOE team on site for a long period
• PERT Review FY06
• The review will cover all aspects of procurement.
  
• The criteria include the following
• BSA has a system in place to use capable and
  reliable contractors who either have track
  records of successful past performance or can
  demonstrate a current superior ability to perform
• Technical evaluations are adequate and support
  source selection decisions.

6
Discussion/Background

• The Real Question S/B
• Do we need to evaluate the supplier beforehand?
  i.e. Is the supplier capable of providing the
  items / services that meet all requirements
  (e.g. technical, quality, cost and
  schedule)?
• This activity is primarily designed to identify
  and reduce the risks of doing business with an
  unknown (uncertain) source!
• BEST PRACTICE!

7
Improvement Opportunity

• Former SA more restrictive than necessary
• The QA Order Rule does NOT specify that an
  evaluation of the QA Program is required
• Many pre-award evaluations are being performed
  which may not include the suppliers QA Program
  (e.g. JNCP, PEP)
• Revised SA allows for these other types of
  evaluations (e.g. technical, environmental,
  financial, etc.), while meeting the intent of the
  DOE Order Rule requirement.

8
Improvement Opportunity

• Implement a web-based system for recording
  evaluations (SES Supplier Evaluation System)
• allows for sharing of information and potential
  cost avoidance (evaluated suppliers good for 3
  years)
• the individual or group making the final
  supplier/ contractor selection is responsible for
  conducting and recording the evaluation results
  in the SES
• will be utilized to ensure evaluations are
  conducted PRIOR to placing PO.

9
Requirements

• Three requirements added
• Pre-Award Evaluations required for Non-commercial
  procurements (not from a catalog not
  off-the-shelf) over 100K since a financial risk
  exists.
• NOTE Since these types of procurements generally
• require some type of evaluation, the impact
  should be
• minimal.
• Evaluations must be conducted by a qualified
  evaluator - the individual or group conducting
  the evaluation who possesses knowledge of the
  evaluation criteria and experience in conducting
  supplier evaluations using the method selected.
• Evaluation Summary must be recorded in BNLs
  Supplier Evaluation System (SES)

10
Supplier Evaluation Summary

• Evaluation of a suppliers capability is required
  for ALL Noncommercial (e.g. custom BNL specs.)
  procurements
• that are risk-significant (either A1Critical or
  A2Major)
• OR
• greater than 100K, UNLESS
• The Supplier has a documented, acceptable
  evaluation within
• the last 3 years (hence the need to SHARE
  evaluation
• information in SES!)
• For Commercial (off-the-shelf/catalog) purchases
  that are risk-significant (A1/A2)
• Documented acceptance criteria and
  inspection/testing is required (in lieu of an
  evaluation) 4-Way Match Rule

11
Supplier Evaluation Methods

• Several methods can be used to perform an
  evaluation
• Third Party (e.g. ISIS Evaluation, ISO 9001
  Registration)
• Desk-top Review
• Survey, Technical Evaluation, Supplier History
• JNCP (sole source justification)
• Proposal Evaluation Panel (PEP) report
• On-site Assessment / Audit
• There is NO substitute for an on-site evaluation!

least effective
most effective
12
The New Web Requisition
13
Supplier Evaluation Systems

• A BNL Supplier evaluation must be documented in
• Supplier Evaluation System (SES)
• Web-based, central repository for evaluations.
• http//130.199.76.90/ppm/ses/ses.asp
• Follow-up on any deficiencies is necessary if
  award is to be made.
• Another Source of Evaluated Suppliers
• Integrated Supplier Information System - ISIS
• https//omniweb.ornl.gov/ISIS/

14
Conclusion

• SA Title changed to Supplier Evaluation
• https//sbms.bnl.gov/sbmsearch/subjarea/12/12_SA.
  cfm?parentID12
• Subject Area had wide-spread review input
• Comments solicited from large small science,
  support organizations and PPM.
• To be presented to PPM, Quality Reps. and all
  Department Administrative Staff
• Implemented in SBMS mid-October 2005

15
OTHER

• ISIS Lead Auditor Refresher Training ?
• 1-day, mid-December (12/15th or 16th)
• Web Req. QR Checklist ?
• ?