Meta-Analysis and Strategy Research Dan R. Dalton Kelley School of Business Indiana University

2 A [Very] Brief History of Research Synthesis Averaging Correlations? Combining Significance Levels? The Narrative Review (aka “Counting” Review) Gene Glass (1976) “Invents” Meta-Analysis Early Critics – “An Exercise in Mega-Silliness”

3 An Example of Meta-Analysis (Data Are Simulated) Research Question: The Extent to which Equity Holdings by CEOs Are Related to Firms’ Financial Performance Proposed Moderator: Expected that this Relationship Will be Moderated by the “Maturity of the Firm” (i.e., Firms that Are Five or Less Years Post-IPO vs. Other) Studies Available for Meta-Analysis = 30 (10 are not significant, 10 are positive and significant, 10 are negative and significant)

4 An Example of Meta-Analysis (Data Are Simulated) R = Reliability RR = Range Restriction M = Moderator (1 = ≤ 5 yrs. Post-IPO; 2 = > 5 yrs. Post-IPO) rnRyRxRRyRRxM

5 “r” - A Bivariate Correlation “r” vs. “d” R-square Deriving “r” from “d,” “t,” “F-score,” “Z,” “Chi-Square” … “r” from Incomplete Information r = Z/sqrt n if “n” = 120 and Z = 1.96 with “r” unknown then r = +/ (i.e., 1.96/10.95)

6 “r” – A Bivariate Correlation, cntd. -17 to +17 and Enter What? Discard the Study? “r” and the Z-transformation? “r” and Statistical Significance And, a “Surprise” About Multiple Non- Significant Results

7 “r” – A Bivariate Correlation and “n” “r” As an Independent Variable, a Dependent Variable, a Control Variable, a Moderating Variable, a Mediating Variable… “n” – The Sample Size from which the “r” Was Calculated To Weight the Observed Correlation in Order to Calculate the Mean Weighted Correlation Across All of the Studies “n” and the Correlation Matrix

8 Ry (Reliability of y); Rx (Reliability of x) Constructs vs. Observed Variables Strategic Management Meta-Analyses with Ry = 1 and Rx = 1 Strategic Management Variables Are Not That Good The Choice of Ry and Rx Levels Is Counterintuitive – Lower Ry’s and Rx’s Will Improve the “Corrected r” Ry and Rx at.8

9 RRy & RRx (Range Restriction of y and x) Analytical Issues of Range Restriction Have Become Increasingly Complex In Strategic Management – RRy and RRx as Deliberate Selectivity in the Sample Strategic Management and “Survival” Issues

10 Moderation in Meta-Analysis In Meta-Analysis a “Moderator” Is a Subgroup Profligate Testing for Moderators  Capitalization on Chance  Loss of Statistical Power Moderators Need Not Always Be Operationalized as a Dichotomy

11 Meta-Analytic Procedures and Results PART 1: # of Correlations Combined Sample Size Mean True Score Correlation Std Dev: Mean True Score Correlation Entire Sample 309, Moderation: ≤ 5 Yrs. from IPO 162, Moderation: > 5 Yrs. from IPO 147,

12 Meta-Analytic Procedures and Results PART 2: Mean True Score Correlation 80% Credibility Interval 90% Confidence Interval % Variance Attributable To Artifacts Entire Sample : : Moderation: ≤ 5 Yrs. from IPO : : Moderation: > 5 Yrs. from IPO : :

13 Meta-Analytic Results: Some Diagnostics The Magnitude of the Mean True Score Correlation Does the 90% Confidence Interval Include Zero? Suggests that the Mean True Score Is Not Significant Does the 80% Credibility Interval (Difference between Low and High Estimates) Exceed.11? Suggests the Existence of a Moderator Does the % Variance Attributable to Artifacts Exceed 75%? Suggests that a Moderator Is Unlikely And, If the Tests Had Relied on Different Rx and Ry Values? [.7 =.48;.8 =.417;.9 =.37 ]

14 Results Summary There is no simple relationship ( -.026, ns) between CEO equity holdings and firm financial performance. There is, however, some evidence of the existence of a moderating variable. There is evidence of a moderating effect for time since IPO. The relationship between CEO equity holdings and firm financial performance for firms 5 years or less from IPO is.417, a significant relationship. The diagnostics suggest that a further moderating effect of this result is unlikely.

15 Results Summary, cntd. The relationship between CEO equity holdings and firm financial performance for firms more than 5 years from the IPO is -.144, a significant relationship. The diagnostics suggest that a further moderating effect of this result is likely.

16 Other Issues in Meta-Analysis Fixed vs. Random Effects Models  Random Effects Models – Population Parameters May Vary Across Studies  Fixed Effects Models – Population Parameters Are Invariant “File Drawer” Problem Unreported Null Results “Fail Safe” Approach The Issue Is Less a Matter of Fail Safe Algorithms than of Reliance on Too Few Studies

17 Other Issues in Meta-Analysis, cntd. Quality of Data Outliers  Statistical Outliers  Entry Error Outliers Sensitivity to Outliers The General Question of Discarding Data Disclosure and Replicability

18 Other Issues in Meta-Analysis, cntd. The Independence of Data Entering Data that Are Clearly Not Independent A Random Selection, Pooling, a Weighted “r”, a Weighted “n” An Interesting Catch-22 “Clearly Reflect the Same Construct” Independence of Samples  Constructive Replication

19 General Guidelines for Meta-Analysis There is no need to transform the input values of “r”s. When it is necessary to impute the value of “r,” set “r” = 0. For observed variables, rely on.8 for the reliability of the dependent and independent variables. With observed variables, it will rarely be necessary to assign a range restriction score.

20 General Guidelines for Meta-Analysis, cntd. Use a conservative 90% confidence interval for the meta-analysis diagnostics (for these data, 95% would be an interval of to.075, much wider than the to.059 reported). Use a conservative 80% credibility interval for the meta-analysis diagnostics (for these data, the 90% would have been an interval of to.448, much wider than the to.336 reported). Where the meta-analysis software provides an option, rely on a “Random Effects Model.”

21 General Guidelines for Meta-Analysis, cntd. Assuming every effort has been made for an exhaustive search for meta-analysis input data, you need not be concerned about “file drawer” issues Neither weight nor exclude data on the basis of the quality of the study. Instead, run two meta-analyses and compare the results for the entire data set and a reduced data set without the troublesome data

22 General Guidelines for Meta-Analysis, cntd. Only under extremely rare conditions would there be any concerns about the independence of the data; accordingly, there is no need to combine data from separate “r”s in any manner. No need to exclude outliers. Instead, run two meta-analyses and compare the results for the entire data set and a data set without the outliers.