International Standards for Survey Quality:

1
International Standards for Survey Quality  A
Total Survey Error Solution?

• Paul P. Biemer
• Lars E. Lyberg

2
AsideEstimator of the MSE Using a Gold Standard
is biased
is unbiased (gold standard)
See equation 2.20, Chapter 2, Overview of Design
Issues Total Survey Error in Marsden, P. and
Wright, J. (2010). Handbook of Survey Research,
Emerald Publishers
3
The Premise of this Paper

• There are many standards for surveys examples
  include
• ISO 20252 for Market, Opinion and Social Research
• OMB standards
• NCES statistical standards
• Quality guidelines developed by specific
  organizations (Stat Can RTI etc.)
• ESS Code of Practice
• Do these standards help in reducing total survey
  error for surveys?
• The answer to this question is not clear.

4
Definitions policies, standards, guidelines and
recommended practices.

• standards - A practice that is followed almost
  without exception. Deviations are not
  recommended and require approval of senior
  management. Corrective action should be taken
  when a standard is not being met.
• policies (no exceptions under any circumstances)
• guidelines (can be skipped for good reasons),
  and
• recommended practices (promoted but not
  obligatory)
• Colledge and March (1997)

5
What are survey standards?

• A document that
• describes methods and procedures for the
  collecting, processing, storing, and presenting
  survey data.  
• define the (minimal) level of quality and effort
  that is acceptable for all survey processes
• May address only some survey processes e.g.,
  standards for writing survey reports
• Should be auditable vague standards have no
  effect on survey quality

6
What purposes do survey standards serve?

• Communicates a level of quality that
  organizations establish as minimal for all
  surveys.
• Provide consistency across surveys in different
  organizations
• Facilitate communication of complex concepts,
  formulas, procedures and methodologies
• Provide transparency of the methodologies used to
  produce a survey data set.
• Transfers skills and knowledge of best survey
  practice

7
Some Problems with Survey Standards

• Comprehensive standards can overwhelm survey
  practitioners
• Unless mandated and audited, they are largely
  ignored.
• Must be continually updated.
• May conflict with client requirements.
• Can be costly to develop, maintain and enforce.
• No documented evidence exists that they improve
  data quality.

8
Idea advocated in this presentation

• General survey standards have their strengths and
  weaknesses, but their effects on data quality
  have not be documented
• Instead we advocate Standards for Survey Quality
  Investigations (SSQIs)
• SSQIs are like survey standards in that they
  describe methods and procedures.
• SSQIs focus on methodological studies and survey
  quality investigations.
• Example the AAPOR standard for computing
  response rates.

9
What purposes would SSQIs serve?

• To promote data quality investigations.
• To encourage sharing of the results of survey
  methodological studies, including the data
  themselves.
• To communicate the methods and the quality
  metrics used.
• To facilitate methodological meta-analysis and
  comparative analysis
• To increase transparency and consistency
• To share skills and knowledge regarding survey
  quality investigations.
• Decrease process variation

10
Some areas where SSQIs would be beneficial

• Computation of frequently used quality indicators
• weighted and unweighted response rates (e.g.,
  AAPOR)
• eligibility rates
• tracing rates
• frame coverage, duplication, and erroneous
  inclusion rates
• design effects (unequal weighting and clustering)
• keying, coding and editing error rates
• Item nonresponse rates
• reliability measures

11
Some areas where SSQIs would be beneficial
(contd)

• Reporting standards for these quality indicators
• Best methods for collecting data to compute these
  quality indicators
• Best practices for
• Nonresponse followup studies, including two phase
  sampling methods
• Investigations of interviewer variance
• Evaluations of frame coverage bias
• Reinterview studies
• Experimentation with incentives

12
Resurrecting an old ITSEW idea Survey Quality
Wiki

• Wiki Hawaiian for fast backronymed to
  stand for what I know is
• Website that allows the easy creation and editing
  of any number of interlinked web pages via a web
  browser using a simplified text editor
• Typically powered by wiki softwareto create
  collaborative websites for knowledge management.
• Users use their editorial rights to remove
  material that is considered "off topic." Such is
  the case of the collaborative encyclopedia
  Wikipedia.

13
Questions for Discussion

• How does the group feel about survey standards in
  general?
• How well do they serve their stated purposes?
• Are there good examples of cases where they have
  improved survey quality?
• How does the group feel about SSQIs? Would such
  standards benefit survey quality research?
• What types of SSQIs would be the most beneficial
  for survey research?
• How should SSQIs be promulgated? Is a Survey
  Quality Wiki a good idea? Is there interest in
  developing it?