[Jdm-society] Increasing the Percentage of Papers Replicated: What about the Denominator?

[Terry Connolly]

HI everyone:
         Scott's proposal is a variant on one I proposed to the ASQ 
Board years ago: That all papers submitted would be published, and 
reviewers would evaluate papers by awarding page length (a practice 
we follow coarsely already, as in "Cut it down to a research note and 
we'll take it" (minor value) or "This is important; Can you develop 
it at monograph length and we'll give you the whole issue" (Slovic 
and Lichtenstein, 1972, say, in OBHP). But here's the trick: The 
minimum publication will contain (a) the author(s)' full names and 
(b) the first word of the title. Obviously, this will produce quite a 
listing of useless papers in the back of each issue, nicely 
alphabetized so you can quickly check which of your friends sent in a 
piece of junk this month -- who could resist making that item the 
first thing we'd scan in each issue? There'd be a mildly useful 
group, published at title-plus-abstract length, so interested folks 
would know who was doing what and maybe write for a copy. There's be 
some at research note length; and a few published in full. To keep 
one's name off the embarrassing junk ("Terry Connolly: The...") list, 
we'd only send in really good papers -- which, I think, is what we 
collectively wish for: fewer but better. And all done by 
self-censoring. Replications would fit nicely into the 
brief-but-useful category, I'd guess, since theory and method 
sections would simply refer to the paper being replicated.
         Terry

Terry Connolly
FINOVA Professor of Management and Organizations
The Eller College
University of Arizona.At 07:36 AM 5/10/2007, armstrong wrote:
>Increasing the Percentage of Papers Replicated: What about the Denominator?
>
>In the interesting JDM List discussion on replications, the focus 
>was on increasing the number of replications. Indeed, we did this in 
>our paper when we suggested that:
>
>o       authors should provide footnotes to direct readers to data 
>and methods (in enough detail to permit direct replication) on the 
>Internet, and
>o       editors of journals should
>o       invite replications of important papers,
>o       evaluate research proposals for replications with an eye to 
>their subsequent publication,
>o       appoint replications editors, and
>o       publish all competent replications.
>
>Another approach is to decrease the denominator. One reason there 
>are few replications is that there are few papers worth replicating. 
>When I ask colleagues to estimate the percentage of published 
>academic papers that have real or potential value, they typically 
>provide quite small percentages. I have made empirical estimates of 
>useful papers in forecasting and in marketing, and in each case the 
>estimate was no higher than 3%.  (See my paper, "Does an Academic 
>Research Paper Contain Useful Knowledge? No (p <. .05)" in full text 
>at http://jscottarmstrong.com ).
>
>As I have argued elsewhere*, one way to reduce the number of papers 
>published is for journals to publish all papers submitted. If 
>anything can be published, promotion committees would be foolish to 
>consider the number of papers published. So researchers who have 
>nothing important to report would have no motive to publish.
>
>  What does it mean to publish all papers submitted? Papers would be 
> reviewed and the reviews would be published along with the paper. 
> Authors would be given the opportunity to revise or withdraw their 
> paper in light of the reviews. Given Internet capabilities, this 
> policy can be carried out at a low cost. Signed moderated open peer 
> review would be published alongside the papers at any time (as is 
> done for products sold on Amazon), and authors would have an 
> opportunity to respond.
>Would anyone publish? Of course! Those with important findings would 
>publish. Universities might then be led to promote people who have 
>important findings that stand up under continuous peer review, and 
>whose papers are read and deemed important enough to be cited, 
>replicated, and used.
>* e.g., J. Scott Armstrong, 2003, "The Value of Surprising Findings 
>for Research on Marketing", Journal of Business Research, 2003, 
>91-92 in full text at http://jscottarmstrong.com
>
>--
>J. Scott Armstrong
>Professor of Marketing, 747 Huntsman, The Wharton School, U. of PA, 
>Phila, PA 19104
>http://www.jscottarmstrong.com
>home phone 610 622 6480
>Home address: 645 Harper Ave., Drexel Hill, PA 19026
>Fax at school: 215 898 2534
>_______________________________________________
>Jdm-society mailing list
>Jdm-society at mail.sjdm.org
>http://www.sjdm.org/mailman/listinfo/jdm-society

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.sjdm.org/mail-archive/jdm-society/attachments/20070510/4d6ff0b1/attachment.htm