[Jdm-society] Fwd: Re: Increasing the Percentage of Papers Replicated: What about the Denominator?

[Jack Dowie]

This has been extremely rewarding but I'm wondering if there is anything
relevant to be said by those of us who see most science as mainly
(only?) an input into decision. Doesn't Value of Information Analysis
have something to say about the value of replication and wouldn't it
suggest there is no general answer to the question being addressed here,
only an answer within a  specific decision with specific loss functions?
Not answer that will advance careers and grants of course but....
Jack

Jack Dowie
Emeritus Professor of Health Impact Analysis
Public Health and Policy Dept
London School of Hygiene and Tropical Medicine
Keppel St
London  WC1E  7HT
Phone :[44] (0)20 7927 2034
Fax: [44] (0)20 7580 4524
http://www.lshtm.ac.uk/pehru/staff/jdowie.html
map showing location of LSHTM is at
http://www.lshtm.ac.uk/location/
>>> Eric R Stone <estone at wfu.edu> 05/11/07 10:59 PM >>>
This has been a most interesting thread, and I have very much enjoyed
reading such a range of thoughtful posts on the subject. I am less
optimistic than some that publishing all papers would ultimately
increase
the quality of those posted, but I do see some value to that proposal.
Personally, I suspect that Maya is right, that after seeing the
difficulties inherent in a system that published all submitted article
along with the reviews, someone would suggest something like the present
system.  I also suspect that after much initial enthusiasm for the new
(i.e., present) system, it would end up getting shelved, since we
(obviously) can't have a system that doesn't provide a mechanism for
publishing important work that wouldn't make it in such a system (e.g.,
replications and non-replications).

I don't have a strong opinion about what the best solution is, but I do
want to add my voice to the call for a vehicle for making replications
and
non-replications more available.  For all of our statistical
sophistication, I have been struck by how easy it is to treat a
significant result as "real" and a non-significant result as "not
there,"
however much we know better.  I have also been struck by the number of
times I (or a colleague) have failed to replicate a result, contacted
the
original authors, only to find out that the original authors have
abandoned their approach (usually couched in phrases like we are now
using
a "more robust manipulation").  It may well not be worth devoting
present
journal space to providing information about replications, but some
mechanism that easily provided access to this type of information for
people looking for it would, I think, save a lot of time in the long
run.
For example, if there were some way of tagging articles with links to
"attempts to replicate" as well as the current information (e.g., number
of citations) that would be very helpful for people interested in
investigating a particular question in detail.

Thanks for reading,
Eric

On Fri, 11 May 2007, armstrong wrote:

> >Dear Maya,
> >
> >Current technology can solve the problem you raise. Instead of
> >censorship by reviewers, a reader could merely click to sort papers
> >by quality (e.g., as judged by original reviewers,  by readers,  by
> >number of citations, by the amount of time spent by readers, or by
> >whether they have been replicated/extended). And given that we are
> >reducing the value of a publications for authors, there would be
> >fewer papers that need to be sorted.
> >
> >Scott
> >
> >>Hi Scott,
> >>
> >>>  What does it mean to publish all papers submitted? Papers would
be
> >>>  reviewed and the reviews would be published along with the paper.
> >>>  Authors would be given the opportunity to revise or withdraw
their
> >>>  paper in light of the reviews. Given Internet capabilities, this
> >>>  policy can be carried out at a low cost. Signed moderated open
peer
> >>>  review would be published alongside the papers at any time (as is
> >>>  done for products sold on Amazon), and authors would have an
> >>>  opportunity to respond.
> >>
> >>I had to smile when I read your proposal.  It might (or might not)
achieve
> >>your goal, but the cost! the cost!  In fact, if the status quo in
the
> >>world of academic publishing were what you are proposing, things
would be
> >>so dismal, I fear, that to relieve them, someone would have invented
the
> >>... present system (i.e., if all papers were published, alongside
referee
> >>reports, somebody would have come up with the brilliant idea of only
> >>publishing those that got favorable reviews).
> >>
> >>Maya Bar-Hillel
> >
> >
> >--
> >
> >J. Scott Armstrong
> >Professor of Marketing, 747 Huntsman, The Wharton School, U. of PA,
> >Phila, PA 19104
> >http://www.jscottarmstrong.com
> >home phone 610 622 6480
> >Home address: 645 Harper Ave., Drexel Hill, PA 19026
> >Fax at school: 215 898 2534
>
>
> --
> J. Scott Armstrong
> Professor of Marketing, 747 Huntsman, The Wharton School, U. of PA,
> Phila, PA 19104
> http://www.jscottarmstrong.com
> home phone 610 622 6480
> Home address: 645 Harper Ave., Drexel Hill, PA 19026
> Fax at school: 215 898 2534
_______________________________________________
Jdm-society mailing list
Jdm-society at mail.sjdm.org
http://www.sjdm.org/mailman/listinfo/jdm-society