European Conference on Quality in Survey Statistics

1
European Conference on Quality in Survey
Statistics
2
Quality in Official StatisticsSome Recent and
Not so Recent Developments

• Lars Lyberg
• Statistics SwedenQ2006

3
Why We Have a Q Conference

• One of the LEG recommendations
• The ESS mission, where it is stated that ESS
  shall provide the EU and indeed the world, with
  high quality information, available to everyone,
  on various areas and levels for decision-making,
  research and debate.
• The ESS vision with keywords such as world
  leader, scientific principles, continuous
  improvement, harmonization, and basis for
  democracy and progress.
• Westat, Inc.

4
Contents of Q so Far ( ms)

• Evaluation of data quality (92)
• Sampling and estimation (65)
• Nonresponse (44)
• Questionnaire development and testing (22)
• Confidentiality (17)
• Burden (5)
• Knowledge economy (5)

• Quality management of systems and organizations
  (63)
• Frameworks (11)
• Reporting (52)
• Process control (36)
• Auditing and self-assessment (11)
• Customers (36)
• Standards (11)
• Harmonization (12)

5
Somewhat Neglected Topics

• Respondents
• Costs
• Trade-offs
• Standardization
• Fitness for use
• User perception of quality
• Trust
• Audits and self-assessment
• The nitty-gritty of QM

6
Issue No.10The Concept of Quality

• Statistical Process Control (30s and 40s)
• Small errors indicate usefulness (Kendall,
  Jessen, Palmer, Deming, Stephan, Hansen, Hurwitz,
  Tepping, Mahalanobis)
• Decomposition of MSE around 1960
• Data quality (Kish, Zarkovich 1965)
• Quality frameworks 70s
• CASM movement 80s
• Quality and users
• The UN Fundamental Principles

7
Components of Quality
8
Defining Quality

• Fitness for use or fitness for purpose
• Framework components
• Getting the job done, on time, within budget, so
  that it meets the specified requirements

9
Quality Assurance and Quality Control

• QA makes sure that the processes are capable of
  delivering a good product
• QC makes sure that the product is actually good

10
Controlling Quality
Measures Indicators
Control instrument
Main stake-holders
Quality Level
Framework dimensions, error est., MSE
Product specs
User
Product
Variation via control charts, other paradata
analysis
Process variables, SPC, CBM, SOP , checklists
Survey designer
Process
Scores, Strong and weak points, Are we measuring
up?
Excellence models, CoP, Reviews, Audits,
Self-assessments
NSI, owner, society
Organization
11
Issue No. 9Quality Measurement and Quality
Reporting

• Objective To ensure that users have access to
  measures or indicators of quality, presented in
  ways that meet their particular needs
• The typical framework relevance, accuracy,
  timeliness and punctuality, accessibility and
  clarity, comparability, and coherence

12
Examples of Reports

• Dataset-specific quality assessments for
  different kinds of economic statistics (IMF)
• Process data handbook (LEG/UK)
• Quality guidelines (Stats Canada, Stats Finland)
• Questions and Answers (OMB)
• National or organizational frameworks
• Quality profiles
• Guidelines for quality reporting (Stats Can,
  ONS,, Stats Sweden, FCSM)

13
Concerns

• The user has not been consulted
• How should dimensions be measured?
• How do we handle information gaps?

• Some quality indicators are dubious
• Dimensions are in conflict
• What happened to total survey error or total
  quality? Särndal and Platek (2001)
• Do we need global harmonization?

14
Issue No. 8Demings 13 points

• The 13 factors that affect the usefulness of a
  survey
• To point out the need for directing effort toward
  all of them in the planning process with a view
  to usefulness and funds available
• To point out the futility of concentrating on
  only one or two of them
• To point out the need for theories of bias and
  variability that correlate accumulated experience

15
The 13 Points

1. Variability in Response
2. Differences between Different Kinds and Degrees
   of Canvass
3. Bias and Variation Arising from the Interviewer
4. Bias of the Auspices
5. Imperfections in the Design of the Questionnaire
   and Tabulation Plans

16
13 Points Continued

1. Changes that Take Place in the Universe before
   Tabulations Are Available
2. Bias Arising from Nonresponse
3. Bias Arising from Late Reports
4. Bias Arising from an Unrepresentative Selection
   of Data for the Survey or of the Period Covered
5. Bias Arising from an Unrepresentative Selection
   of Respondents

17
13 Points Continued

1. Sampling Errors and Biases
2. Processing Errors
3. Errors in Interpretation

18
Issue No. 7The Race for the No.1 Spot

• Started with The Economists ranking
• There is an element of positioning in some of the
  visions presented by statistical organizations

19
But

• There is no justification for competition
• There is no framework, jury or reward
• Statistical organizations have the same problems
  and tasks and need to collaborate
• Statistical organizations should capitalize on
  their strengths and develop excellence centre
  networks and share knowledge

20
Global Coordination

• Kotz (2005) The statistical community is
  witnessing an astonishing lack of coordination
  between many hundreds of statistical offices and
  agencies scattered throughout the world.
• ..without an overall planning, some of the
  efforts of civil servants and researchers are
  largely wasted.
• well-planned international measures are urgent.
• new basic global definitions of basic concepts
  need to be developed

21
Issue No. 6Quality Management

• TQM, Business reengineering, Balanced scorecard,
  business excellence models, Six Sigma
• Tools and core values
• Aversion to QM acronyms
• The management principles cannot be used
  uniformly across countries and companies
• Operations vs research culture
• Culture eats strategy for breakfast
• We are left with a set of very useful tools and
  work principles

22
Examples

• The process view
• Key process variables, paradata, control charts
• Spirit of continuous improvement
• Extensive user involvement
• Adoption of the PDCA cycle
• The importance of leadership
• Organizing work, inspiration, focussing on
  important issues, going for root cause,
  benchmarking, developing staff competence,
  evaluating approaches used, promoting good
  examples, empowerment, communication

23
From Good to GreatJim Collins

• Whats so special with businesses that have
• been very successful for at least 15 years?
• Level 5 leadership
• First who, then what
• Confront the brutal facts, yet never lose faith
• The hedgehog concept
• Culture of discipline

24
Issue No. 5Competence

• Staff competence
• Excellent programs within the U.S. Federal System
  (JPSM, USDA)
• Stats Canada, INSEE, ONS, ABS
• Excellent university programs
• User competence

25
Competence Issues

• Existing programs heavy on methodology
• Sampling and estimation
• Software
• Specialization
• Not much on broader aspects of quality
• Many NSIs talk about the need to skill up
• Any examples of vigorous attempts vis-a-vis the
  user?

26
Issue No. 4 Comparative Studies

• Comparative studies are increasingly
• important
• Short term economic indicators
• Literacy surveys
• Social surveys (EU-SILC, ESS)
• Education surveys

27
Examples of challenges

• Existing systems for input and output
  harmonization are not sufficient
• Developing a questionnaire that works in all
  countries and languages
• Concepts, questions, translation, interpretation
• Extensive quality control and supervision
• Varying methodological and financial resources
• Increased distance between user and producer

28
Issue No. 3The Process View

• Traditional large-scale evaluations are expensive
  and results come too late
• Small-scale evaluations must be conducted to get
  estimates of error components (gold standard,
  latent class analysis, responsive designs,
  multi-level modelling)
• Long-term improvements are achieved via improved
  processes controlled by paradata

29
Generic Control Chart
30
Understanding Variation (I)

• Common cause variation
• Common causes are the process inputs and
  conditions that contribute to the regular,
  everyday variation in a process
• Every process has common cause variation
• Example Percentage of correctly scanned data,
  affected by peoples handwriting, operation of
  the scanner

31
Understanding Variation (II)

• Special cause variation
• Special causes are factors that are not always
  present in a process but appear because of
  particular circumstances
• The effect can be large
• Special cause variation is not present all the
  time
• Example Using paper with a colour unsuitable for
  scanning

32
Action

• Eliminate special cause variation
• Decrease common cause variation if necessary
• Do not treat common cause as special cause

33
Standards

• Purposes
• To control processes, variability and costs
• To improve comparability
• To define a minimum level of performance
• Examples
• Classification
• CBMs and checklists
• Standard Operating Procedures
• ISO

34
Problems with Standards

• They must be adhered to
• They must be maintained and updated
• In stovepipe systems its easy to find excuses to
  deviate
• Standard, policy, guideline, best practice,
  recommended practice?

35
Issue No. 2The User

• In place
• The principle of openness (OMB 1978)
• Responsibility to inform users (many agencies in
  the 70s)
• Dissemination procedures
• Customer satisfaction and image surveys
• Councils and service level agreements

• Problems
• How should quality information be communicated?
• How do we distinguish between different kinds of
  users?
• How do users and producers use quality
  information and metadata?
• How do producers and users collaborate on fitness
  for use? (ABS)

36
Issue No. 1Image Is Everything
37

1. Eliminate special cause variation
2. Decrease common cause variation if necessary

38
European Conference on Quality in Survey
Statistics