Recent Quality Related Initiatives at Statistics Canada

1
Recent Quality Related Initiatives at Statistics
Canada

• Claude Julien
• Statistics Canada
• Quality Secretariat
• Q2008, Rome

2
Introduction

• Statistics Canadas Quality Assurance Framework
• Research on adding another component to the
  framework presented at Q2006
• Soon after, a few unfortunate errors led to three
  quality related initiatives

3
Outline

• Introduction
• Review of quality assurance practices
• Quality assurance learning and awareness exercise
• Additional quality checks and analysis in the
  data release step

4
Review of quality assurance practices in
statistical programs
5
Design

• Objectives
• Identify quality risks to address
• Identify quality best practices to share
• Scope
• Operational checks throughout the production
  process
• Analytical checks to validate statistical
  information
• Data release operations
• Focus on production, not design
• Focus on processes, not outputs

6
Design (cont.)

• Coverage - Nine mission critical programs
• Assessment approach
• Concurrent internal independent review
• Three reviewers per program
• Three programs per reviewer

7
Findings

• Good overall approach
• Sound QA practices in most production steps of
  most programs
• More common risk factors affecting programs
• Loss of expertise
• Lack of staff / staff turnover
• Timeliness pressures on monthly programs
• Complexity of data release operations

8
Outcome

• Report disseminated to the public
• Allocation of additional resources to address
  more urgent needs
• Development of ongoing review program integrated
  into existing planning, priority setting and
  review mechanisms
• Input into other initiatives

9
Quality assurance learning and awareness exercise
10
Design

• Main objectives Bring survey managers together
  to discuss QA practices in their programs
  (learning)
• Other objectives
• Surface main areas in their programs at risk of
  causing significant errors (risk awareness)
• Provide managers with basis to deal with risks
  (take action risk management)
• Identify areas that are more at risk collectively

11
Design (cont.)

• E-learning exercise (computer based training)
• Group exercise
• Discuss and answer over 100 questions on QA
  practices gathered from
• existing production and risk management practices
• input from quality review (means of sharing best
  practices)

12
Outcome

• Broad coverage in very short time over 800
  participants in less than 4 months
• Useful discussion and exchange of opinions among
  participants
• Presence/absence of over 100 QA practices in 80
  statistical programs

13
Findings

• Most of the risk lies among certain types of QA
  practices (e.g. documentation) rather than in
  certain types of programs
• Support for the scope of the reviews on quality
  assurance practices
• Need to review learning and development strategy
  on QA, project management and risk management

14
Quality checks and analysis in the data release
step
15
Complexity of data release

• Increased relevance of information
• Increased timeliness and accessibility
• 5 releases per day on average
• Release article, data tables, publications and
  metadata all at once
• Numerous systems with many interdependencies
• Many people involved from many parts of the
  organization

16
Recent initiatives

• Additional checks
• Moving checks upstream
• Systematic recording and analysis of corrections
  made to articles just prior to or after release
  (i.e. get the facts)
• Feedback to managers
• Regular reporting to senior management

17
Recording and analysis of corrections

• Characteristics
• When (month and frequency of release)
• Where in article (text, tables, charts or
  metadata)
• What in article (words, numbers, reference
  period, link, etc.)
• How important (potential impact on quality)
• Underlying causes and factors of more important
  corrections
• Where in survey process did the error occur?
• Why did the error occur?

18
Findings

• Many corrections do not affect accuracy
• Nearly all corrections after release are made on
  the day of release (within a few hours)
• Errors earlier in the survey process are less
  likely, but have higher potential impact
• Errors in the data release steps are more likely,
  but have lower potential impact
• Some suggestion of seasonality

19
Summary
20
Quality Assurance Framework - Before

• Design
• User consultation
• Subject matter consultation
• Subject matter training
• Quality Guidelines
• Standards

• Production
• Knowledge, experience, skill
• Project management
• Quality Control

21
Quality Assurance Framework - Now

• Design
• User consultation
• Subject matter consultation
• Subject matter training
• Quality Guidelines
• Standards

• Production
• Knowledge, experience, skill
• Project management
• (More) Quality Control (in data release step)
• Quality reviews
• Quality learning and awareness
• Risk management

22
Outcome
23
Grazie! / Merci!

• For more information, or
• to obtain a French copy
• of this presentation,
• please contact

• Pour de plus amples
• informations ou pour obtenir
• une copie en français du
• document, veuillez
• contacter

Claude Julien Email / Courriel
claude.julien_at_statcan.ca Phone number /
Téléphone 613-951-6937