[Jdm-society] Increasing the Percentage of Papers Replicated: What about the Denominator?

[armstrong]

Increasing the Percentage of Papers Replicated: What about the Denominator?

In the interesting JDM List discussion on replications, the focus was 
on increasing the number of replications. Indeed, we did this in our 
paper when we suggested that:

o	authors should provide footnotes to direct readers to data 
and methods (in enough detail to permit direct replication) on the 
Internet, and
o	editors of journals should
o	invite replications of important papers,
o	evaluate research proposals for replications with an eye to 
their subsequent publication,
o	appoint replications editors, and
o	publish all competent replications.

Another approach is to decrease the denominator. One reason there are 
few replications is that there are few papers worth replicating. When 
I ask colleagues to estimate the percentage of published academic 
papers that have real or potential value, they typically provide 
quite small percentages. I have made empirical estimates of useful 
papers in forecasting and in marketing, and in each case the estimate 
was no higher than 3%.  (See my paper, "Does an Academic Research 
Paper Contain Useful Knowledge? No (p <. .05)" in full text at 
http://jscottarmstrong.com ).

As I have argued elsewhere*, one way to reduce the number of papers 
published is for journals to publish all papers submitted. If 
anything can be published, promotion committees would be foolish to 
consider the number of papers published. So researchers who have 
nothing important to report would have no motive to publish.

  What does it mean to publish all papers submitted? Papers would be 
reviewed and the reviews would be published along with the paper. 
Authors would be given the opportunity to revise or withdraw their 
paper in light of the reviews. Given Internet capabilities, this 
policy can be carried out at a low cost. Signed moderated open peer 
review would be published alongside the papers at any time (as is 
done for products sold on Amazon), and authors would have an 
opportunity to respond.

Would anyone publish? Of course! Those with important findings would 
publish. Universities might then be led to promote people who have 
important findings that stand up under continuous peer review, and 
whose papers are read and deemed important enough to be cited, 
replicated, and used.

* e.g., J. Scott Armstrong, 2003, "The Value of Surprising Findings 
for Research on Marketing", Journal of Business Research, 2003, 91-92 
in full text at http://jscottarmstrong.com

-- 
J. Scott Armstrong
Professor of Marketing, 747 Huntsman, The Wharton School, U. of PA, 
Phila, PA 19104
http://www.jscottarmstrong.com
home phone 610 622 6480
Home address: 645 Harper Ave., Drexel Hill, PA 19026
Fax at school: 215 898 2534
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.sjdm.org/mail-archive/jdm-society/attachments/20070510/5adfc6a3/attachment-0001.htm