Maria Joo Zilho

1

Statistical Auditing and Self-Assessment
Benefits for Quality Improvement

• Maria João Zilhão
• Planing and Quality Control Unit

Q 2008 / Rome July 2008
2
Concepts .
Self-Assessment
Self-assessment is a comprehensive, systematic
and regular review of an organizations
activities and results referenced against a
model/framework
An audit is a systematic, independent and
documented process for obtaining audit evidence
and evaluating it objectively to determine the
extent to which the audit criteria are fulfilled
Audit
3
The Importance of Auditing and Self-assessment in
statistics
The Leadership Group on Quality
A generic checklist should be developed for a
simple self-assessment programme for survey
managers
DESAP
The methods for auditing on different levels and
for diferent purposes such as internal, external
,.., should be reiviewd and recommendations
provided
Handbook on experiences on auditing
European Statistics Code of Practice
4.2 4.5
Principle 4 Quality Commitment
4
The DatQAM map(Handbook on Data Quality
assessment Methods and Tools Eurostat / 2007)
External Environment of DatQAM
User Requirements
Standards
Labelling
Certification (ISO 20252)
Self-Assessment
Audits
Process Variables
Quality Indicators
Quality Reports
User Surveys
Measurement of Processes and Products
Precoditions of DatQAM
5
References for auditing

• ISO Norms
• 90002000Implementation of Quality Management
  Systems
• 19011 - .. Auditing
• 10013 - Documentation systems

6
References for self-assessment

• EFQM Excellence Model
• (European Foundation for Quality Management)
• CAF Common Assessment Framework (EIPA -
  European Institution for Public Administration)
• DESAP Checklist for survey Managers
• (Eurostat)

7
Preconditions
8
Quality Management System at Statistics Portugal
Quality Documentation System
ISO 19011
Departments
ISO 10013
Documenting Processes
Departments
Departments
Suggestions / Complaints
Internal Audits
Customer Satisfaction Surveys
External Audits
Performance indicators
Quality Management Unit
COP Principle 4
9
Documentation System
Documenting processes
ISO 10013
Procedures handbooks
Documentation on survey processes

• Process design
• Definition of different phases of the process
• Definition of responsibilities
• Definition of minimum documentation
• Handbooks are produced by pluridisciplinary teams

10
Statistical Production Handbook Procedures



Phases
IV. Evaluation
III. Dissemination
II. Operation
I. Conception
Processes
Processes
Process
Process
II.1. Planing and preparation of data collection
I.1. Viability Study
III.1. Dissemination
IV.1. Evaluation
I.2. Methodological Study
II.2. Data collection
II.3. Data treatment and analysis
Audits
I.3. Technical approval
Note Each Process is composed by sub-processes
and tasks
11
Internal Quality Audits at Statistics Portugal
(I)
Objectives
To verify the degree of procedures implementation
(handbooks procedures) To Identify improvement
opportunities To check the internal
articulation/communication of units (survey
related) To Improve survey documentation
12
Internal Quality Audits at Statistics Portugal
(II)



Statistical Production Handbook Procedures
IV. Evaluation
III. Dissemination
II. Operation
I. Conception
II.1. Planing and preparation of data collection
I.1. Viability Study
III.1. Dissemination
IV.1. Evaluation
I.2. Methodological Study
II.2. Data collection
II.3. Data treatment and analysis
I.3. Technical approval

13
Internal Quality Audits at Statistics Portugal
Preparation (III)
Training of 14 Auditors (Statisticians) (ISO
19011)
Discussion
Top Management Support
Exchange of Information / Strong Communication
about auditing, through Top Management, and the
Quality Unit
Internal Procedure on auditing (Clear rules!)
14
Internal Quality Audits at Statistics Portugal
(IV)
Principal rules
Top Management defines the Quality/Statistical
Audits annual plan No one should audit its own
activity One Audit team needs 2 or 3
auditors Auditors produce Audit Plans and reports
according to Procedures Improvement actions must
be carried out by the audited departments Improvem
ent actions must be published and followed by the
audit teams
15
External Quality Audits at Statistics Portugal

• Auditors are Peers (from other statistical
  Offices)
• Basic rules are the same as for internal
  auditing
• Two visits (/- one week each)
• Contacts between auditors and audited teams in
  the mid period
• Auditors work recommendations with audited teams
• Results are presented to the board
• Action plan is made by audited teams and
  presented to the board

16
Benefits for Quality Improvement
The identification of improvement opportunities
in the audited process The implementation of
corrective / preventive actions The redefinition
of procedures Auditing involves people in the
quality program (auditors and audited teams) Good
Recommendations - A look from the
outside! Identification of good practices that
can be spread in the organization Self-assessment
and auditing can be combined together A time to
re-think and improve!
Some NSIs have experienced Quality Auditing!
A Powerfull Tool!

• BUT!!!!
• -Time consuming activity
• May face resistances
• May be dificult to start
• - Better results in the long run if
  systematically implemented

17
THANK YOU! Maria João Zilhão Statistics
Portugal Planing and Quality Control
Unit mjoao.zilhao_at_ine.pt