[Jdm-society] on the need for replications

[John Sterman]

Replication studies in economics have had a similar fate as those  
Scott Armstrong cites in marketing .  Hubbard and Vetter (1991, 1992)  
examined more than 2700 empirical papers in economics and finance.   
They found only one pure replication.  Only about 4% of journal pages  
were devoted to replications with extensions of the original work.

Notwithstanding the issues of regression to the mean and low power  
that may cause replication failure even when effects are real, the  
need for replication is great simply to find errors in methods and  
data reduction.  Dewald, Thursby, and Anderson (1986) worked with the  
Journal of Money, Credit and Banking to replicate empirical studies  
received for consideration.  They found one-third of the authors of  
previously published papers did not respond to repeated requests to  
supply their data sets.  Half the remainder chose not to submit their  
data or could not locate the data.  Response rates were much higher  
for those whose papers were under consideration or just published  
(for obvious reasons).  Among those who did respond, Dewald et al.  
found 69 instances of error in or inability to replicate results in  
54 data sets.  They conclude, “Inadvertent errors in published  
empirical articles are a commonplace rather than a rare occurrence”  
and cautioned that “The frequency and magnitude of errors in  
empirical articles raise serious questions regarding the integrity of  
the refereeing process of professional journals.”  In a follow-up  
article, Anderson and Dewald (1994) were disappointed to find that “A  
decade after the JMCB project, the replication of previous studies as  
a part of new research seems an infrequent occurrence.”

Hubbard, R. and D. Vetter (1991) Replications in the finance  
literature:  An empirical study, Journal of Business and Economics 30 
(4), 70-81.

Anderson, R. and W. Dewald (1994) Replication and scientific  
standards in applied economics a decade after the Journal of Money,  
Credit and Banking project, Federal Reserve Bank of St. Louis Review  
76(6), 79-83.

Dewald, W., J. Thursby, and R. Anderson (1986) Replication in  
empirical economics, the Journal of Money, Credit, and Banking  
Project, American Economic Review 76(4), 587-603.



John Sterman

Jay W. Forrester Professor of Management
Director, MIT System Dynamics Group
Sloan School of Management
30 Wadsworth Street, Room E53-351
Cambridge MA 02142
617.253.1951 (voice)
617.258.7579 (fax)
jsterman at mit.edu
http://web.mit.edu/jsterman/www



>
> _______________________________________________
> Jdm-society mailing list
> Jdm-society at mail.sjdm.org
> http://www.sjdm.org/mailman/listinfo/jdm-society