[Jdm-society] Replications

[Eran Magen]

I am absolutely loving this thread, and am so glad to see this topic
discussed openly and earnestly. The problem of failed replications has
a strong negative effect on the field, since the experimental
community seems to "canonize" published results, and consider any
failed replications to be faulty - even if results were reported just
once, or only by the same author time and again. There are a number of
well-known and oft-cited experiments which seem to never be replicated
outside of the original publishing group. Reporting results that
cannot be reproduced (perhaps because the original outcome happened by
chance alone, or because the methods are under-specified) is what
prevents psychology from becoming a truly incremental science.

In the spirit of brainstorming possible solution, here's an idea that
I've been discussing with friends and colleagues over the past year or
so, as this topic has been on my mind.

1) As a requirement for publication, certain journals would require a
"seal of replicability" (the process for receiving one is described
below). I imagine a journal that only published studies which have
been proven to be replicable by a third party would garner a *lot* of
attention and regard.
2) Upon receiving a submission and following inital triage, the editor
would send the manuscript and any accompanying materials to a central
replication center (which can be coordinated by the APA, NSF, NIH...
or multiple agencies could exist for different fields of study).
3) The center would either carry out the replication study by itself
(using in-house researchers and facilities) or outsource this task
(e.g. by sending it to participating MA programs, as suggested earlier
in this thread). The study will be designed by following the "methods"
section of the manuscript (and any supporting online materials), and
using any accompanying material as stimuli.
4) Successful replications would be reported back to the journal (via
the coordinating center), which would then proceed with the
publication process.


* A few obvious objections to this method:

1) This method won't work for all types of studies. What if an
experiment used very uncommon equipment (e.g. life-sized elephant
robots) or very expensive procedures (e.g. fMRI)? This would require
too much effort to replicate.

- True. We can start by replicating studies that *are* easy to
conduct, such as the common stare-and-click studies, or others that do
not require expensive procedures or materials. If there is an unusual
set of stimuli involved in the process, the author can send those in
(well, perhaps not life-sized elephant robot, but definitely any
unusual visual stimuli, audio files, etc.).

2) The publication process will be slowed down.

- True, it would. However, not by much, if the author provides all of
the necessary stimuli and a clear methods section. The replication
centers, after all, will be churning those things out, and so will
have lots of participants and dedicated researchers standing by. This
can even happen as the reviewers go over the paper, to save time.

3) Operating these replication centers will be expensive, and nobody
will give the money.

- We are losing much more (money and credibility) by NOT having such
replication centers. Tremendous amounts of resources are given to
researchers who report sexy-yet-irreplicable studies, and the entire
discipline (and, to be just a little idealistic, humanity at large) is
constantly paying the price for supporting a non-incremental science.
Implementing a system of this sort will guarantee a much more
thoughtful, careful, and robust system of experimentation.

4) What if the replication center will be using a different population
than the original experimenter?

- It's up to the experimenter to specify whether or not she believes
that the results are generalizable, or whether they only pertain to
specific parts of the population. There's no reason to believe that a
replication center (or any of its collaborators) will have any trouble
with finding participants. Again, this can only lead to more careful
claims, and more thoughtful discussions of results.

5) What if lots of studies won't be replicated?

- Good! This will push us to take ourselves seriously, and take steps
to become a truly incremental science. Perhaps we'll need to change
criteria for significance. We'll probably need to start being very
specific about our methods, which will naturally lead to less wild
claims about their general applicability.

If You've read this Email this far - hat's off, and thanks for taking
the time. :-) I'm glad (as usual) to be a part of this wonderful JDM
group, which is willing to take on an issue as delicate and as crucial
as this. I believe good things will come out of this discussion.

Take care, enjoy the meanwhile,

- Eran Magen


@-`-,-'--

Have You visited http://www.TheHungerSite.com today? You really get to
send food to hungry people, and it really only costs You 5 seconds and
a click.


On 5/6/07, mdoher2 at bgnet.bgsu.edu <mdoher2 at bgnet.bgsu.edu> wrote:
> My first week as a faculty member at Bowling Green was in September of 1965. The Ohio
> Board of Regents had approved our PhD program in August. That first week was a week-long
> faculty meeting, dedicated almost exclusively to putting the meat onto the bones of the
> approved doctoral proposal. For a few years after that, I kept promoting the idea that an
> acceptable Master's thesis should be a literal replication, to the extent permitted by the
> quality of the Method section of the paper that had reported the experiment being replicated.
> To the best of my recollection, not a single person agreed! I still think it is a good idea, and it
> would not create another requirement (I comletely concur with Jon's terror of adding yet
> another requirement) and it would likely get the students into an active research mode much
> sooner.
>
> There is a problem. A couple of years later, I taught a seminar in cognitive, and had as a
> requirement that the students select an important study that could be replicated with the
> resources at hand, but they only had to run a single subject. The purposes of doing so were
> to get them to read more closely than they ever had before, and to get them to see the
> imprtance of writing clear and complete method sections. Every student concluded that they
> could not do a truly literal replication without contacting the author(s) of the paper!
>
> Perhaps stressing the need for replication might have a side effect of more complete
> writeups?
>
> Mike Doherty
>
>
> _______________________________________________
> Jdm-society mailing list
> Jdm-society at mail.sjdm.org
> http://www.sjdm.org/mailman/listinfo/jdm-society