False (and Missed) Discoveries in Financial Economics
Corresponding Author
CAMPBELL R. HARVEY
Correspondence: Campbell Harvey, Duke University and National Bureau of Economic Research; e-mail: cam.harvey@duke.edu
Search for more papers by this authorYAN LIU
Campbell R. Harvey is with Duke University and National Bureau of Economic Research. Yan Liu is with Purdue University. We appreciate the comments of Stefan Nagel; two anonymous referees; as well as Laurent Barras, Claude Erb, Juhani Linnainmaa, and Michael Weber; and seminar participants at Rice University, University of Southern California, University of Michigan, University of California at Irvine, Hanken School of Economics, Man-Numeric; Research Affiliates; and the 2018 Western Finance Association meetings in San Diego. Kay Jaitly provided editorial assistance. The authors do not have any potential conflicts of interest, as identified in the JF Disclosure Policy.
Search for more papers by this authorCorresponding Author
CAMPBELL R. HARVEY
Correspondence: Campbell Harvey, Duke University and National Bureau of Economic Research; e-mail: cam.harvey@duke.edu
Search for more papers by this authorYAN LIU
Campbell R. Harvey is with Duke University and National Bureau of Economic Research. Yan Liu is with Purdue University. We appreciate the comments of Stefan Nagel; two anonymous referees; as well as Laurent Barras, Claude Erb, Juhani Linnainmaa, and Michael Weber; and seminar participants at Rice University, University of Southern California, University of Michigan, University of California at Irvine, Hanken School of Economics, Man-Numeric; Research Affiliates; and the 2018 Western Finance Association meetings in San Diego. Kay Jaitly provided editorial assistance. The authors do not have any potential conflicts of interest, as identified in the JF Disclosure Policy.
Search for more papers by this authorABSTRACT
Multiple testing plagues many important questions in finance such as fund and factor selection. We propose a new way to calibrate both Type I and Type II errors. Next, using a double-bootstrap method, we establish a t-statistic hurdle that is associated with a specific false discovery rate (e.g., 5%). We also establish a hurdle that is associated with a certain acceptable ratio of misses to false discoveries (Type II error scaled by Type I error), which effectively allows for differential costs of the two types of mistakes. Evaluating current methods, we find that they lack power to detect outperforming managers.
Supporting Information
| Filename | Description |
|---|---|
| jofi12951-sup-0001-InternetAppendix.pdf325.6 KB |
Appendix S1: Internet Appendix. |
| jofi12951-sup-0002-ReplicationCode.zip16.1 KB |
Replication code. |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
REFERENCES
- Andrikogiannopoulou, Angie, and Filippos Papakonstantinou, 2016, Estimating mutual fund skill: A new approch, Working paper, Swiss Finance Institute.
- Andrikogiannopoulou, Angie, and Filippos Papakonstantinou, 2019, Reassessing false discoveries in mutual fund performance: Skill, luck, or lack of power? Journal of Finance 74, 2667–2688.
- Avramov, Doron, Robert Kosowski, Narayan Y. Naik, and Melvyn Teo, 2011, Hedge funds, managerial skill, and macroeconomic variables, Journal of Financial Economics 99, 672–692.
- Ayadi, Mohamed A., and Lawrence Kryzanowski, 2011, Fixed-income fund performance: Role of luck and ability in tail membership, Journal of Empirical Finance 18, 379–392.
- Bajgrowicz, Pierre, and Olivier Scaillet, 2012, Technical trading revisited: False discoveries, persistence tests, and transaction costs, Journal of Financial Economics 106, 473–491.
- Barras, Laurent, 2019, A large-scale approach for evaluating asset pricing models, Journal of Financial Economics 134, 549–569.
- Barras, Laurent, Olivier Scaillet, and Russ Wermers, 2010, False discoveries in mutual fund performance: Measuring luck in estimated alphas, Journal of Finance 65, 179–216.
- Barras, Laurent, Olivier Scaillet, and Russ Wermers, 2018, Reassessing false discoveries in mutual fund performance: Skill, luck, or lack of power? A reply. Working paper, McGill University.
- Beneish, Messod D., 1997, Detecting GAAP violation: Implications for assessing earnings management among firms with extreme financial performance, Journal of Accounting and Public Policy, Fall 1997, 271–309.
10.1016/S0278-4254(97)00023-9 Google Scholar
- Beneish, Messod D., 1999, The detection of earnings manipulation, Financial Analysts' Journal 55, 24–36.
10.2469/faj.v55.n5.2296 Google Scholar
- Benjamini, Yoav, and Yosef Hochberg, 1995, Controlling for the false discovery rate: A practical and powerful approach to multiple testing, Journal of the Royal Statistical Society, Series B 57, 289–300.
10.1111/j.2517-6161.1995.tb02031.x Google Scholar
- Benjamini, Yoav, and Daniel Yekutieli, 2001, The control of the false discovery rate in multiple testing under dependency, Annuals of Statistics 29, 1165–1188.
- Berk, Jonathan B., and Richard C. Green, 2004, Mutual fund flows and performance in rational markets, Journal of Political Economy 112, 1269–1295.
- Blake, David, Alberto Rossi, Allan Timmermann, Ian Tonks, and Russ Wermers, 2013, Decentralized investment management: Evidence from the pension fund industry, Journal of Finance 68, 1133–1178.
- Brown, S. J., W. Goetzmann, R. G. Ibbotson, and S. A. Ross, 1992, Survivorship bias in performance studies, Review of Financial Studies 5, 553–580.
- Busse, Jeffrey A., Amit Goyal, and Sunil Wahal, 2014, Investing in a global world, Review of Finance 18, 561–590.
- Carhart, Mark M., 1997, On persistence in mutual fund performance, Journal of Finance 52, 57–82.
- Carhart, M.M., J.N. Carpenter, A.W. Lynch, and D.K. Musto, 2002, Mutual fund survivorship, Review of Financial Studies 15, 1439–1463.
- Cao, Charles, Yong Chen, Bing Liang, and Andrew W. Lo, 2013, Can hedge funds time market liquidity? Journal of Financial Economics 109, 493–516.
- Chen, Joseph, Harrison Hong, Ming Huang, and Jeffrey D. Kubik, 2004, Does fund size erode mutual fund performance? The role of liquidity and organization, American Economic Review 94, 1276–1302.
- Chen, Yong, and Bing Liang, 2007, Do market timing hedge funds time the market? Journal of Financial and Quantitative Analysis 42, 827–856.
- Chordia, Tarun, Amit Goyal, and Alessio Saretto, 2020, Anomalies and false rejections, Review of Financial Studies 33, 2134–2179.
- D'Agostino, Antonello, Kieran McQuinn, and Karl Whelan, 2012, Are some forecasters really better than others? Journal of Money, Credit, and Banking 44, 715–732.
- De Long, J. Bradford, and Kevin Lang, 1992, Are all economic hypotheses false? Journal of Political Economy 100, 1257–1272.
- DeGroot, Morris, 1975, Probability and Statistics (Addison-Wesley, Reading, MA).
- DeGroot, Morris, and Mark J. Schervish, 2011, Probability and Statistics, 4th edition (Pearson Education Limited, Harlow, UK)
- Diane Del, Guercio,, and Paula A. Tkac, 2008, Star power: The effect of Morningstar ratings on mutual fund flow, Journal of Financial and Quantitative Analysis 43, 907–936.
- Elton, Edwin J., Martin J. Gruber, and Christopher R. Blake, 1996, Survivor bias and mutual fund performance, Review of Financial Studies 9, 1097–1120.
- Elton, Edwin J., Martin J. Gruber, and Christopher R. Blake, 2001, A first look at the accuracy of the CRSP mutual fund database and a comparison of the CRSP and Morningstar mutual fund database, Journal of Finance 56, 2415–2430.
- Fama, Eugene F., and Kenneth R. French, 1993, Common risk factors in the returns on stocks and bonds, Journal of Financial Economics 33, 3–56.
- Fama, Eugene F., and Kenneth R. French, 2010, Luck versus skill in the cross-section of mutual fund returns, Journal of Finance 65, 1915–1947.
- Fawcett, T., 2006, An introduction to ROC analysis, Pattern Recognition Letters 27, 861–874.
- Ferson, Wayne, and Yong Chen, 2017, Working paper, University of Southern California.
- Genovese, Christopher, and Larry Wasserman, 2002, Operating charateristics and extensions of the false discovery rate procedure, Journal of the Royal Statistical Society, Series B 64, 499–517.
- Giglio, Stefano, Yuan Liao, and Dacheng Xiu, 2018, Thousands of alpha tests, Working paper, University of Chicago.
- Hastie, T., Robert Tibshirani, and Jerome Friedman, 2009, The Elements of Statistical Learning: Data Mining, Inference, and Prediction (Springer Science & Business Media) New York, New York.
10.1007/978-0-387-84858-7 Google Scholar
- Harvey, Campbell R., 2017, Presidential address: The scientific outlook in financial economics, Journal of Finance 72, 1399–1440.
- Harvey, Campbell R., and Yan Liu, 2013, Multiple testing in economics, Working paper, Duke University.
- Harvey, Campbell R., Yan Liu, and Heqing Zhu, 2016, … and the cross-section of expected returns, Review of Financial Studies 29, 5–72.
- Harvey, Campbell R., and Yan Liu, 2017, Luck vs. skill and factor selection, in John Cochrane and Tobias J. Moskowitz, eds.: The Fama Portfolio (University of Chicago Press, Chicago, IL).
- Harvey, Campbell R., and Yan Liu, 2018, Detecting repeatable performance, Review of Financial Studies 31, 2499–2552.
- Harvey, Campbell R., and Yan Liu, 2019, Lucky factors, Working paper, Duke University.
- Harvey, Campbell R., and Yan Liu, 2020, Revisiting luck vs. skill in mutual fund evaluation, Working paper, Duke University.
- Harvey, Campbell R., Yan Liu, Nicholas G. Polson, and Jianeng Xu, 2019, Re-evaluating semi-strong market efficiency, Working paper, Duke University.
- Hau, Harald, and Sandy Lai, 2013, Real effects of stock underpricing, Journal of Financial Economics 108, 392–408.
- Horowitz, Joel L., 2001, The Bootstrap, Handbook of Econometrics 5 (Elsevier, Amsterdam, The Netherlands).
- Hou, Kewei, Chen Xue, and Lu Zhang, 2020, Replicating anomalies, Review of Financial Studies 33, 2019–2133.
- Ioannidis, John P. A., 2005, Why most published research findings are false? PLoS Medicine 2, e124.
- Ioannidis, John P. A., and Chris H. Doucouliagos, 2013, What's to know about the credibility of empirical economics, Journal of Economic Surveys 27, 997–1004.
- Ioannidis, John P. A., Tom D. Stanley, and Hristos Doucouliagos, 2017, The power of bias in economics research, Economic Journal 127, 236–265.
- Jiang, George J., Tong Yao, and Tong Yu, 2007, Do mutual funds time the market? Evidence from portfolio holdings, Journal of Financial Economics 86, 724–758.
- Jones, Christopher S. and Jay Shanken, 2005, Mutual fund performance with learning across funds, Journal of Financial Economics 78, 507–552.
- Kandel, Shmuel, and Robert F. Stambaugh, 1996, On the predictability of stock returns: An asset-allocation perspective, Journal of Finance 51, 385–424.
- Kosowski, Robert, Allan Timmermann, Russ Wermers, and Hal White, 2006, Can mutual fund “stars” really pick stocks? New evidence from a bootstrap analysis, Journal of Finance 61, 2551–2595.
- Leamer, Edward E., 1983, Let's take the con out of econometrics, American Economic Review 73, 31–43.
- Linnainmaa, Juhani T., 2013, Reverse survivorship bias, Journal of Finance 68, 789–813.
- Nanda, Vikram, Z. Jay Wang, and Lu Zheng, 2004, Family values and the star phenomenon: Strategies of mutual fund families, Review of Financial Studies 17, 667–698.
- Romano, Joseph P., Azeem M. Shaikh, and Michael Wolf, 2008, Control of the false discovery rate under dependence using the bootstrap and subsampling, TEST 17, 417–442.
- Romano, Joseph P., and Michael Wolf, 2005, Stepwise multiple testing as formalized data snooping, Econometrica 73, 1237–1282.
- Sarkar, Sanat K., 2006, False discovery and false nondiscovery rates in single-step multiple testing procedures, Annuals of Statistics 34, 394–415.
- Sastry, Ravi, 2013, The cross-section of investing skill, Working paper, University of Melbourne.
- Scott, James G., and James O. Berger, 2006, An exploration of aspects of Bayesian multiple testing, Journal of Statistical Planning and Inference 136, 2144–2162.
- Storey, John D., 2002, A direct approach to false discovery rates, Journal of the Royal Statistical Society, Series B 64, 479–498.
- Storey, John D., 2003, The positive false discovery rate: A Bayesian interpretation and the q-value, Annals of Statistics 31, 2013–2035.
- Yan, Xuemin, and Lingling Zheng, 2017, Fundamental analysis and the cross-section of stocks returns: A data-mining approach, Review of Financial Studies 30, 1382–1423.
- Ziliak, Stephen T., and Deirdre N. McCloskey, 2004, Size matters: The standard error of regressions in the American Economic Review 33, 527–546.
