Critical Finance Review > Vol 7 > Issue

Is Economics Research Replicable? Sixty Published Papers From Thirteen Journals Say “Often Not”

Andrew C. Chang, Board of Governors of the Federal Reserve System, USA, a.christopher.chang@gmail.com Phillip Li, Office of Financial Research, U.S. Department of the Treasury, USA, phillip.li@ofr.treasury.gov
 
Suggested Citation
Andrew C. Chang and Phillip Li (2018), "Is Economics Research Replicable? Sixty Published Papers From Thirteen Journals Say “Often Not”", Critical Finance Review: Vol. 7: No. . http://dx.doi.org/10.1561/104.00000053

Forthcoming: 01 Oct 2018
© 2018 A. C. Chang and P. Li
 
Subjects
 
Keywords
B41C80C82C87C88E01
Data and code filesGross domestic productNational income and product accountsPreanalysisPublication biasReplication
 

Article Help

Share

Download article
In this article:
1. Introduction
2. Methodology and Sample
3. Summary of Results
4. What About Trying Harder to Obtain Replication Files?
5. Heterogeneity in Paper Characteristics and Replication Success
6. Conclusion and Recommendations
References

Abstract

We attempted to replicate 67 macroeconomic papers published in 13 well-regarded economics journals using author-provided replication files that included both data and code by following a preanalysis plan. Aside from six papers that used confidential data, we obtained data and code replication files for 29 of 35 papers (83%) that were required to provide such files as a condition of publication, compared to 11 of 26 papers (42%) that were not required to provide data and code replication files. Defining replication success as our ability to use the author-provided data and code files to produce the key qualitative conclusions of the original paper, we successfully replicated 22 of 67 papers (33%) without contacting the authors. Excluding the six papers that used confidential data and the two papers that used software we did not possess, we replicated 29 of 59 papers (49%) with assistance from the authors. Because we were able to replicate less than half of the papers in our sample even with help from the authors, we assert that economics research is often not replicable. We conclude with recommendations on improving replication of economics research.

DOI:10.1561/104.00000053