[Jdm-society] Fwd: Re: Increasing the Percentage of Papers Replicated: What about the Denominator?

[Eric R Stone]

This has been a most interesting thread, and I have very much enjoyed
reading such a range of thoughtful posts on the subject. I am less
optimistic than some that publishing all papers would ultimately increase
the quality of those posted, but I do see some value to that proposal.
Personally, I suspect that Maya is right, that after seeing the
difficulties inherent in a system that published all submitted article
along with the reviews, someone would suggest something like the present
system.  I also suspect that after much initial enthusiasm for the new
(i.e., present) system, it would end up getting shelved, since we
(obviously) can't have a system that doesn't provide a mechanism for
publishing important work that wouldn't make it in such a system (e.g.,
replications and non-replications).

I don't have a strong opinion about what the best solution is, but I do
want to add my voice to the call for a vehicle for making replications and
non-replications more available.  For all of our statistical
sophistication, I have been struck by how easy it is to treat a
significant result as "real" and a non-significant result as "not there,"
however much we know better.  I have also been struck by the number of
times I (or a colleague) have failed to replicate a result, contacted the
original authors, only to find out that the original authors have
abandoned their approach (usually couched in phrases like we are now using
a "more robust manipulation").  It may well not be worth devoting present
journal space to providing information about replications, but some
mechanism that easily provided access to this type of information for
people looking for it would, I think, save a lot of time in the long run.
For example, if there were some way of tagging articles with links to
"attempts to replicate" as well as the current information (e.g., number
of citations) that would be very helpful for people interested in
investigating a particular question in detail.

Thanks for reading,
Eric

On Fri, 11 May 2007, armstrong wrote:

> >Dear Maya,
> >
> >Current technology can solve the problem you raise. Instead of
> >censorship by reviewers, a reader could merely click to sort papers
> >by quality (e.g., as judged by original reviewers,  by readers,  by
> >number of citations, by the amount of time spent by readers, or by
> >whether they have been replicated/extended). And given that we are
> >reducing the value of a publications for authors, there would be
> >fewer papers that need to be sorted.
> >
> >Scott
> >
> >>Hi Scott,
> >>
> >>>  What does it mean to publish all papers submitted? Papers would be
> >>>  reviewed and the reviews would be published along with the paper.
> >>>  Authors would be given the opportunity to revise or withdraw their
> >>>  paper in light of the reviews. Given Internet capabilities, this
> >>>  policy can be carried out at a low cost. Signed moderated open peer
> >>>  review would be published alongside the papers at any time (as is
> >>>  done for products sold on Amazon), and authors would have an
> >>>  opportunity to respond.
> >>
> >>I had to smile when I read your proposal.  It might (or might not) achieve
> >>your goal, but the cost! the cost!  In fact, if the status quo in the
> >>world of academic publishing were what you are proposing, things would be
> >>so dismal, I fear, that to relieve them, someone would have invented the
> >>... present system (i.e., if all papers were published, alongside referee
> >>reports, somebody would have come up with the brilliant idea of only
> >>publishing those that got favorable reviews).
> >>
> >>Maya Bar-Hillel
> >
> >
> >--
> >
> >J. Scott Armstrong
> >Professor of Marketing, 747 Huntsman, The Wharton School, U. of PA,
> >Phila, PA 19104
> >http://www.jscottarmstrong.com
> >home phone 610 622 6480
> >Home address: 645 Harper Ave., Drexel Hill, PA 19026
> >Fax at school: 215 898 2534
>
>
> --
> J. Scott Armstrong
> Professor of Marketing, 747 Huntsman, The Wharton School, U. of PA,
> Phila, PA 19104
> http://www.jscottarmstrong.com
> home phone 610 622 6480
> Home address: 645 Harper Ave., Drexel Hill, PA 19026
> Fax at school: 215 898 2534