Harmonization by Design

1
Harmonization by Design

• Beth-Ellen Pennell
• Zeina Mneimneh
• European Conference on Quality in Survey
  Statistics
• April 2006
• Cardiff

2
Observations re Comparative, Cross-national,
Cross-cultural Studies

• Links between attitudes, values, norms, and
  behaviors are reinforced if observed across
  cultures and contexts
• Provide natural quasi-experimental design --
  can examine how structure and policy impact
  behavior
• Ex ante harmonization -- study design
  acknowledging cross-cultural implementation --
  done infrequently but increasing in practice

3
Observations Continued

• Many of these studies suffer from inadequate
  design, process control and lack basic
  documentation to assess quality
• Best practices from mono-cultural survey research
  methods have been slow in transferring to
  cross-cultural research
• HOWEVER, cross-cultural research methods are not
  simply mono-cultural methods applied in a
  different context

4
Observations Continued

• Cross-cultural research calls for a unique set of
  procedures, processes, and protocols that can be
  appropriately adapted to local conditions

5
Overview

• Context for complexity
• World Mental Health Survey Initiative
• Optimization versus localization
• When all else fails
• Document, document, document

6
The U.S. National Comorbidity Survey-Replication

• 9836 respondents
• 5774 variables
• 1 language
• Yields 56,793,064 pieces of data

7
World Mental Health Survey Initiative

• 30 countries
• 35 languages
• 312,078 respondents
• 5774 variables
• Yields 1,801,938,372 pieces of data

8
Quantity Impacts Quality

• Volume of data
• Variation in sampling frames
• Survey content, translation and adaptation
• Data collection procedures
• Interviewer recruitment, training and oversight
• Implementation of quality control
• Response rates / non-response bias
• Local norms/contexts

9
WMH Overview

• Motivation
• Principal investigators
• Ronald Kessler, Ph.D., Harvard University
• Bedirhan Üstün, M.D., World Health Organization
• Funding
• U.S. National Institute of Mental Health,
  European Commission, the MacArthur Foundation,
  the Robert Wood Johnson Foundation, World Health
  Organization, Pan American Health Organization,
  various pharmaceutical companies, and the
  governments of the participating countries
• Technical coordination
• Data Collection Coordination Center University
  of Michigan
• Data Analysis Coordination Center Harvard
  University

10
Study Goals

• Study designed to
• measure the prevalence and severity of mental
  disorders
• determine the global burden of mental disorders
• assess service use and the medications used to
  treat mental disorders and
• characterize those who receive treatment, those
  who remain untreated, and identify the barriers
  to treatment.
• Identify common and country-specific risk factors

11
Overview of WMHStudy Design

• Probability sample design (all stages) adults
• 312,078 in-person interviews
• Both computerized (CAPI) and paper versions
  (PAPI)
• 30 countries in all 6 WHO regions Shared design,
  training, quality control, and data processing
  protocols

12
WMH Participating Countries, Sample Size, and
Languages
13
World Mental Health Participating Countries Data
Collection Status
14
Europe
World Map
15
Middle East
World Map
16
Balancing Optimization and Comparability or
Research Imperialism and Collaboration
17
Enforcing Comparability

• Certification Process
• Evidence and documentation at each step of survey
  lifecycle
• Willingness to make the hard decisions

18
Lessons Learned

• Coordination across organizations
• Field organizations with varying degrees of
  survey experience technical expertise
• Difficulty with concept of probability sampling
• Inadequate quality control procedures
• Unfamiliarity with nonresponse reduction
  techniques
• Inadequate interviewer training/oversight
• Inadequate budget detail to assess competency

19
Hindsight

• Delineation of roles, responsibilities and
  authority across organizations
• Standardized procedures/protocols and interviewer
  training
• Use of same software platform across countries
• And, if all else fails.

20
Document!

• Collect sufficiently detailed data about the
  survey design and implementation process to
• Facilitate standardization and cross-cultural
  comparison
• Standard measures/procedures
• DDI compliant XML metadata
• Archive survey information and materials
• Reduce administrative burden
• Facilitate replication

21
Goals

• Also to facilitate
• Monitoring of processes
• Assessment of process quality
• Improvements in methods and measures
• Analysis and correct use of data

22
Survey Metadata Documentation System (SMDS)

• ICPSR and ZUMA collaborative development
• Tool designed to facilitate documentation of
  survey lifecycle (as will version 3 of DDI)
• from initial design
• through data collection
• to post-survey processing and archiving

23
SMDS

• Features
• Supports multiple users simultaneously
• Modularized
• Web-based
• Navigation with built in skip logic
• Data reporting options by country, module, or
  question
• Data extraction to third party software package

24
SMDS Metadata Modules
25
SMDS Modules
Select modules in any order complete in multiple
sessions.
26
General Project Information Module
27
Ethics Review Module
28
Sample Design Module
29
Translation Module
30
Data Collection Module
31
Quality Control Module
32
Dataset Preparation/Final Report Module
33
Dataset Preparation/Final Report Module
34
Data Reporting Cross Country Comparison
35
Data Documentation

• Goals
• To facilitate
• Testing
• Human subjects/ethics review
• Version control/translation documentation
• Comparison of instruments used in data collection
• Comparison of data collection instrument against
  newest version
• Codebook generation/archiving
• Public release data files (with appropriate
  links)

36
Process

• Create output from Blaise instrument (CAPI/PAPI)
• Integrate 'cleaned' SAS files from Harvard
• Add created variables
• Produce comparison grids

37
Instrument Comparison Base Structure
38

39

40

41

Instrument Comparison Page
42
Conclusions

• Challenges and Complexities
• Quantity does impact quality
• Organizing structures are critical
• Must strike correct balance between
  standardization and localization
• Need evidence-based best practices for
  cross-cultural research