Best Meta-Analysis Practice

1
Best Meta-Analysis Practice

• As of Spring 2015

2
Before you start

• Define precisely the research question
• Know how you will analyze the data before you
  collect them
• Are there sufficient studies for
• a reasonable CI for an overall effect?
• power for tests of moderators?
• Do you have time and resources to search, gather
  and code data?

3
Data Collection Analysis 1

• Multiple coders for all or at least some of the
  studies analyze agreement/reliability
• Exhaustive search for grey and unpublished
  studies
• Appropriate model(s) for data analysis
• Random effects
• Nesting or dependencies
• Moderator analyses
• Bias Sensitivity analyses

4
Data Collection Analysis 2

• Coding studies
• Choice of effect size
• Assumptions (e.g., if only p value reported) for
  derivation
• Transformations from one effect size to another
• Code-book description
• Consensus method
• Description of raters
• Rater training, N of raters for each judgment
• Rater Agreement/reliability
• Alpha/intraclass correlation
• Kappa
• Percent agreement
• Test-retest

5
Publication 1

• Title MA? Synthesis?
• Abstract
• Include numbers
• Introduction
• Clear statement of research questions
• Controversy in lit?
• Confirm/refute popular opinion
• Create powerful test, e.g., differential validity
  by race
• Probs with previous metas, e.g., average all
  kinds of teams
• Define constructs
• Hypothesize moderators

6
Publication 2

• Data Collection, Sources
• Conference proceedings/abstracts
• Dissertations
• Electronic databases searched include terms,
  keywords
• Google scholar
• Hand searches of literature a must include
  because you get stuff you dont get in electronic
  databases
• Listservs
• Other sources of documents (e.g., key
  researchers/informants)
• Contact authors directly
• Librarian (hire if you can afford)
• Reference sections of articles
• Unpublished manuscripts
• Websites of authors in the area

7
Publication 3

• What data must you include (your stated analysis
  requires what)?
• Effect size
• Moderator
• Sample size
• Specific means, standard deviations, etc.

8
Publication 4

• Eligibility, Inclusion/Exclusion
• Country
• Language
• Measures allowed
• Characteristics of the measures (e.g., minimum
  number of items, reliability, etc.)
• E.g., NEO vs. big five generic
• Specific measures (e.g,. facet vs. general
  satisfaction)
• Participants (e.g., all male)
• Publications (type included)
• Journal (e.g., APA journal)
• Book vs. article, etc.
• Published vs. not vs. subsequently published
• Study design (e.g., specific control groups,
  repeated measures, etc.)
• Time frame

9
Publication 5

• Report coding efforts replicable? Reliable?
• Report Summary Effects (usually mean)
• Model and weights used
• Anti-bias methods (e.g., d to g)
• Overall mean, CI and PI (k and N desirable)
• REVC and CI (k and N desirable)
• Sensitivity analyses
• Outlier
• Publication
• Other

10
Publication 6

• Report Moderator Tests
• Model categorical, continuous, both
• Significance test result
• Practical test result (show magnitude of effect
  subgroup means, regression coefficients)
• Reduction in REVC, R-square analog
• Power of test?
• Any moderator tests not hypothesized in advance
  should be clearly indicated

11
Publication 7

• Graphs
• Flow chart search thru analysis (desirable)
• Forest plot overall summary
• Possible additional graphs
• Boxplot or stem-leaf if too many ES
• Funnel plot (contour ehanced) trim fill
  publication bias
• Cumulative forest publication bias
• Omit-one forest outliers

12
Publication 8

• Discussion
• Conclusion with respect to initial research
  question results clearly linked to questions
• Meaning of results in terms of study (e.g.,
  Binomial Effect Size Display)
• Future research e.g., other moderators, what are
  gaps based on what is known
• Discussion of credible threats / study limits

13
Publication 8

• Discussion
• Conclusion with respect to initial research
  question results clearly linked to questions
• Meaning of results in terms of study (e.g.,
  Binomial Effect Size Display)
• Future research e.g., other moderators, what are
  gaps based on what is known
• Discussion of credible threats / study limits