Quality Management Assessment at Statistics Canada

1
Quality Management Assessment at Statistics Canada
Claude Julien and Alice Born

• European Conference on Quality in Survey
  Statistics
• April 25, 2006

2
Outline

• Quality Management at Statistics Canada
• Why a Quality Management Assessment?
• The characteristics of a QMA
• Options for a QMA
• The next steps
• Conclusion

3
Quality Management at STC

• Long history of sound QM practices
• Audit on QM in 1997
• Quality Assurance Framework
• Audit on 4 key surveys (self-assessment)
• Recommendations of the Office of the
  Auditor-General (OAG)
• Examine a wider application of enhanced
  self-assessment
• Ensure application of Policy on Informing Users
  of Data Quality and Methodology

4
QM at STC (cont.)

• Quality Assurance Framework
• Quality definition (fitness-for-use, 6
  dimensions)
• Links to policies, standards and guidelines
• Framework for reporting on data quality
• Quadrennial and Biennial Program Reports
• Quality of the outputs
• Integrated Metadatabase (IMDB)
• Description, data sources, methods, data accuracy
  and additional documentation on every product

5
Why a QMA?

• Limitations of Program Reports
• Independent program reports for senior management
• No comprehensive assessment of quality management
  within a survey
• No systematic assessment of all surveys
• Strategic, medium and long-term quality
  improvement

6
Why a QMA? (cont.)

• The QMA is a systematic assessment of surveys
  focussing on the standard set of processes used
  to carry them out
• Objectives
• Raise and maintain awareness on quality at survey
  design and implementation level
• Provide input to Program Reports and Corporate
  Planning
• Respond to future requests from OAG and Treasury
  Board
• Reveal gaps in Quality Assurance Framework

7
Characteristics of a QMA

• Scope
• Collection Strategy
• Review Strategy
• Quality Improvement Strategy

8
Scope

• Units are the survey and programs listed on the
  Integrated Metadatabase (IMDB)
• Traditional surveys, administrative data or
  derived statistics
• All, most or some of the over 380 surveys and
  programs since 2000
• Timeframe (strategic, annual, biennial,
  quadrennial)

9
Collection strategy

• Summary information
• Comprehensive set of detailed questions
• Checklist questionnaire
• Detailed information
• Survey specific report with supporting
  documentation
• In coordination with the IMDB

10
Review Strategy

• Global analysis of summary information
• Assess state of QM at various levels or groups of
  surveys
• Identify (un)favourable trends among surveys or
  on processes across several surveys
• In-depth review of detailed information
• Independent review team
• Assess state of QM at survey level

11
Quality Improvement Strategy

• Data collection
• Immediate feedback
• Raise awareness on quality
• Suggest improvement
• Global analysis of summary information
• Informal comparison
• Input to the program reports
• Input to corporate planning
• In-depth review of detailed information
• Feedback and recommendations
• Formal follow-up

12
Options for a QMA
13
Survey statistics
Admin data statistics
Derived statistics
Some
All
Annual
Biennial
Quadrennial
Strategic
Global analysis
Collection of summary information
IMDB
Divisions
Collection of detailed information
STC
Program Reports
Independent review
Planning
Feedback, Recommendations and Follow-up
14
Survey statistics
Admin data statistics
Some
Audits carried out in the late 1990s
Strategic
Collection of detailed information
Independent review
Feedback and Recommendations
15
Questionnaire

• Collect summary information on the processes
• Easy to answer and avoid overlap with IMDB
• Foster continuous quality improvement
• Quality definition
• Quality management practices
• Traditional surveys (Quality Guidelines)
• Administrative data surveys (Quality Guidelines?)
• Derived statistics (?)
• Analytical products (?)

16
Questionnaire (cont.)

• Checklist questionnaire divided by survey process
  or step
• Examples from Statistics Sweden, Eurostat
  (DESAP), any others (please!)
• Questionnaire design and testing
• Cost, burden and awareness of data quality

17
Survey statistics
Admin data statistics
Derived statistics
All
Option 1 Eurostat (DESAP)
Biennial
Quadrennial
Collection of summary information
Quality improvement
IMDB
18
Survey statistics
Admin data statistics
Derived statistics
Option 2 Global analysis
All
Biennial
Quadrennial
Global analysis
Collection of summary information
IMDB
Divisions
STC
Program Reports
Quality improvement
Planning
19
Survey statistics
Admin data statistics
Derived statistics
Option 3 Statistics Sweden Quality Audits
All
Quadrennial
Collection of summary information
IMDB
Collection of detailed information
Independent review
Feedback, Recommendations and Follow-up
Quality improvement
20
Survey statistics
Admin data statistics
Derived statistics
Option 4 Full package
All
Quadrennial
Global analysis
Collection of summary information
IMDB
Some
Divisions
STC
Strategic
Program Reports
Collection of detailed information
Planning
Independent review
Feedback, Recommendations and Follow-up
21
The Next Steps

• Determine the needs of the organization
• Impact on the Quality Assurance Framework
• Questionnaire design and testing
• Feasibility of a global analysis and
  dissemination
• Costs of the in-depth review

22
Conclusion

• Quality Assurance Framework
• Quality reporting mechanism
• Feasibility of a QMA
• Relevance to the organization
• Design
• Options similar to European experiences
• Interested in knowing more about them