Meta-Analysis and Strategy Research

1
Meta-Analysis and Strategy Research

• Dan R. Dalton
• Kelley School of Business
• Indiana University

2
A Very Brief History of Research Synthesis

• Averaging Correlations?
• Combining Significance Levels?
• The Narrative Review (aka Counting Review)
• Gene Glass (1976) Invents Meta-Analysis
• Early Critics An Exercise in
• Mega-Silliness

3
An Example of Meta-Analysis(Data Are Simulated)

• Research Question The Extent to which Equity
  Holdings by CEOs Are Related to Firms Financial
  Performance
• Proposed Moderator Expected that this
  Relationship Will be Moderated by the Maturity
  of the Firm (i.e., Firms that Are Five or Less
  Years Post-IPO vs. Other)
• Studies Available for Meta-Analysis 30
• (10 are not significant, 10 are positive and
• significant, 10 are negative and significant)

4
An Example of Meta-Analysis(Data Are Simulated)

• R Reliability
• RR Range Restriction
• M Moderator (1 5 yrs. Post-IPO 2 gt 5
  yrs. Post-IPO)

r n Ry Rx RRy RRx M
.26 .39 .37 .29 .23 .11 56 225 192 146 70 325 .8 .8 .8 .8 .8 .8 .8 .8 .8 .8 .8 .8 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2
5
r - A Bivariate Correlation

• r vs. d
• R-square
• Deriving r from d, t, F-score, Z,
  Chi-Square
• r from Incomplete Information
• r Z/sqrt n
• if n 120 and Z 1.96 with r unknown
• then r / - .179 (i.e., 1.96/10.95)

6
r A Bivariate Correlation, cntd.

• -17 to 17 and Enter What?
• Discard the Study?
• r and the Z-transformation?
• r and Statistical Significance
• And, a Surprise About Multiple Non-Significant
  Results

7
r A Bivariate Correlationand n

• r As an Independent Variable, a Dependent
  Variable, a Control Variable, a Moderating
  Variable, a Mediating Variable
• n The Sample Size from which the r Was
  Calculated
• To Weight the Observed Correlation in Order to
  Calculate the Mean Weighted Correlation Across
  All of the Studies
• n and the Correlation Matrix

8
Ry (Reliability of y) Rx (Reliability of x)

• Constructs vs. Observed Variables
• Strategic Management Meta-Analyses with Ry 1
  and Rx 1
• Strategic Management Variables Are Not That Good
• The Choice of Ry and Rx Levels Is
  Counterintuitive Lower Rys and Rxs Will
  Improve the Corrected r
• Ry and Rx at .8

9
RRy RRx (Range Restriction of y and x)

• Analytical Issues of Range Restriction Have
  Become Increasingly Complex
• In Strategic Management RRy and RRx as
  Deliberate Selectivity in the Sample
• Strategic Management and Survival Issues

10
Moderation in Meta-Analysis

• In Meta-Analysis a Moderator Is a Subgroup
• Profligate Testing for Moderators
• Capitalization on Chance
• Loss of Statistical Power
• Moderators Need Not Always Be Operationalized as
  a Dichotomy

11
Meta-Analytic Procedures and Results
PART 1 of Correlations Combined Sample Size Mean True Score Correlation Std Dev Mean True Score Correlation
Entire Sample 30 9,685 -.026 .283
Moderation 5 Yrs. from IPO 16 2,032 .417 .048
Moderation gt 5 Yrs. from IPO 14 7,653 -.144 .188
12
Meta-Analytic Procedures and Results
PART 2 Mean True Score Correlation 80 Credibility Interval 90 Confidence Interval Variance Attributable To Artifacts
Entire Sample -.026 - .389 .336 - .112 .059 5.74
Moderation 5 Yrs. from IPO .417 .354 .479 .396 .437 80.49
Moderation gt 5 Yrs. from IPO -.144 - .386 .098 -.061 -.227 7.26
13
Meta-Analytic ResultsSome Diagnostics

• The Magnitude of the Mean True Score Correlation
• Does the 90 Confidence Interval Include Zero?
  Suggests that the Mean True Score Is Not
  Significant
• Does the 80 Credibility Interval (Difference
  between Low and High Estimates) Exceed .11?
  Suggests the Existence of a Moderator
• Does the Variance Attributable to Artifacts
  Exceed 75? Suggests that a Moderator Is Unlikely
• And, If the Tests Had Relied on Different Rx and
  Ry Values? .7 .48 .8 .417 .9 .37

14
Results Summary

• There is no simple relationship ( -.026, ns)
  between CEO equity holdings and firm financial
  performance. There is, however, some evidence of
  the existence of a moderating variable.
• There is evidence of a moderating effect for time
  since IPO. The relationship between CEO equity
  holdings and firm financial performance for firms
  5 years or less from IPO is .417, a significant
  relationship. The diagnostics suggest that a
  further moderating effect of this result is
  unlikely.

15
Results Summary, cntd.

• The relationship between CEO equity holdings and
  firm financial performance for firms more than 5
  years from the IPO is -.144, a significant
  relationship. The diagnostics suggest that a
  further moderating effect of this result is
  likely.

16
Other Issues in Meta-Analysis

• Fixed vs. Random Effects Models
• Random Effects Models Population Parameters May
  Vary Across Studies
• Fixed Effects Models Population Parameters Are
  Invariant
• File Drawer Problem
• Unreported Null Results
• Fail Safe Approach
• The Issue Is Less a Matter of Fail Safe
  Algorithms than of Reliance on Too Few Studies

17
Other Issues in Meta-Analysis, cntd.

• Quality of Data
• Outliers
• Statistical Outliers
• Entry Error Outliers
• Sensitivity to Outliers
• The General Question of Discarding Data
• Disclosure and Replicability

18
Other Issues in Meta-Analysis, cntd.

• The Independence of Data
• Entering Data that Are Clearly Not Independent
• A Random Selection, Pooling, a Weighted r, a
  Weighted n
• An Interesting Catch-22
• Clearly Reflect the Same Construct
• Independence of Samples
• Constructive Replication

19
General Guidelines for Meta-Analysis

• There is no need to transform the input values of
  rs.
• When it is necessary to impute the value of r,
  set r 0.
• For observed variables, rely on .8 for the
  reliability of the dependent and independent
  variables.
• With observed variables, it will rarely be
  necessary to assign a range restriction score.

20
General Guidelines for Meta-Analysis, cntd.

• Use a conservative 90 confidence interval for
  the meta-analysis diagnostics (for these data,
  95 would be an interval of -.128 to .075, much
  wider than the -.112 to .059 reported).
• Use a conservative 80 credibility interval for
  the meta-analysis diagnostics (for these data,
  the 90 would have been an interval of -.493 to
  .448, much wider than the -.389 to .336
  reported).
• Where the meta-analysis software provides an
  option, rely on a Random Effects Model.

21
General Guidelines for Meta-Analysis, cntd.

• Assuming every effort has been made for an
  exhaustive search for meta-analysis input data,
  you need not be concerned about file drawer
  issues
• Neither weight nor exclude data on the basis of
  the quality of the study. Instead, run two
  meta-analyses and compare the results for the
  entire data set and a reduced data set without
  the troublesome data

22
General Guidelines for Meta-Analysis, cntd.

• Only under extremely rare conditions would there
  be any concerns about the independence of the
  data accordingly, there is no need to combine
  data from separate rs in any manner.
• No need to exclude outliers. Instead, run two
  meta-analyses and compare the results for the
  entire data set and a data set without the
  outliers.