Criteria to assess quality of observational studies evaluating the incidence, prevalence, and risk f

1
Criteria to assess quality of observational
studies evaluating the incidence, prevalence, and
risk factors of chronic diseases

• Minnesota EPC
• Clinical Epidemiology Methods Centre, Ottawa
  Health Research Institute, Ottawa
• Tufts University Medical Center, Boston
• RTI International University of North Carolina,
  Chapel Hill
• Blue Cross and Blue Shield Association, Chicago
• Centers for Disease Control and Prevention,
  Atlanta
• Southern California EPC RAND Corporation, Santa
  Monica
• German Institute for Quality and Efficiency in
  Health Care, Köln, Germany
• Johns Hopkins University, Baltimore

2
Analytical tasks
3
Therapeutic vs. non therapeutic studies
4
Systematic literature review

• Systematic reviews and meta-analyses of
  observational nontherapeutic studies published in
  English in core clinical journals (145)
• Published tools (scales or checklists) for
  quality assessment of observational studies (96)
• Annotated bibliography of the methodological
  literature to identify biases and strategies to
  reduce bias in observational studies

5
Development of Checklists

• Separate evaluation of external and internal
  validity
• Exhaustive range of criteria
• Discriminating reporting and methodological
  quality
• Predefined major and minor flaws
• Study and hypotheses level analyses
• Automated reports
• Templates for the protocols of quality
  evaluations with predefined research specific
  quality standards
• Manual, help files, instructions

6
Collaboration with participating experts

• Criteria evaluation face and content validity
• Pilot reliability testing (10 randomly selected
  articles) overall, by topic, by article, by
  domains of external and internal validity, and by
  quality component
• Discriminant validity testing hypotheses of
  detected differences in quality and reporting vs.
  methodological quality

7
Finalizing checklists

• Checklist for studies of incidence or prevalence
  of chronic diseases (MORE)
• Checklist for studies of risk factors of chronic
  diseases (MEVORECH)
• Synthesis of evidence from the studies with major
  flaws
• Level A exclusion from synthesis
• Level B full review and quality abstraction
• Level C separate limited synthesis if major flaws
  detected

8
Quality criteria

• Incidence or prevalence studies
• Aim of study
• Study design
• External Validity
• 1. Sampling of the subjects by the investigators
• 2. Assessment of sampling bias
• 3. Estimation of sampling bias
• 4. Exclusion rate from the analysis
• 5. Sampling bias is addressed in the analysis
• 6. Subject flow
• Internal Validity
• 1. Source to measure outcomes
• 2. Definition of outcomes
• 3. Measurements of outcomes
• 4. Outcomes in race, ethnic, age, or gender
  subpopulations
• 5. Reporting of outcomes

• Risk factors studies
• Aim of study
• Objectives
• Study design
• External Validity
• 1. Sampling of the subjects by the investigators
• 2. Assessment of sampling bias
• 3. Estimation of sampling bias
• 4. Exclusion rate from the analysis
• 5. Sampling bias is addressed in the analysis
• 6. Subject flow
• Internal Validity
• 1.Source to measure outcomes
• 2. Definition of outcomes
• 3. Measurements of outcomes
• 4. Definition of the exposure
• 5.Measurements of the exposure
• 6. Confounding factors

9
Major flaws

• External validity
• Sampling in clinics for
• incidence in the general
• population
• Sampling of controls from different
• population as cases
• Response rate lt40
• Exclusion of gt10 of eligible subjects
• from the analyses
• Reasons to exclude from the analyses
• differ for exposed and not exposed

• Internal validity
• Severity of the outcome can be
• relevant but not assessed in the
• study
• Non valid methods were obtained to
• measure the outcomes, exposure, or
• confounding factors
• Different methods to measure
• exposure (risk factors, independent
• variable) in cases and controls
• No strategies to reduce bias

10
Minor Flaws

• Internal validity
• Outcomes proxy reported
• Obtained from medical records (data mining) or
  administrative database
• Inter-methods validation
• Subjective judgment of reliability
• Intensity/dose of exposure can be relevant but
  not assessed in the study

• External validity
• Random sampling restricted to geographic area
• Convenient sampling
• The authors did not assess sampling bias
• Sampling bias was not addressed in the analysis
• Sampling of controls from health care related
  sources

11
Example of the report
12
Conclusions

• We propose two checklists for transparent and
  standardized quality assessment in systematic
  reviews and evidence-based reports
• Reasonable validity
• Poor random overall and good component
  reliability in a pilot testing of the articles
  from different topics

13
Future Research with Quality Assessment Using
Proposed Checklists

• Protocols of systematic reviews of nontherapeutic
  observational studies with justified definitions
  of research specific quality standards and
  preplanned reliability testing
• Synthesis of evidence incorporating quality
  assessment (sensitivity of the results to
  quality)
• The evaluation of the level of evidence from
  several observational nontherapeutic studies
• Quality assessment of primary studies should
  improve quality of systematic reviews and
  evidence-based reports