[Jdm-society] Replications

[Todd McElroy]

I agree completely with this point and I believe that it would greatly 
improve the validity (and reliability) of our science. However, I also 
agree with the logistical issues that you rightly point out. I would 
suggest that this process might be more likely if it were to be 
introduced incrementally. Specifically, this could be presented as an 
option to authors or prospective authors. During the review process the 
stimuli could be sent out for “review” as well. At this point you should 
ask why someone would want to take such a risk? I would suggest that a 
replication of the study presented in a paper would certainly seem to 
make the paper more attractive to an editor who might be in limbo about 
the work. I would also like to offer another important issue that should 
be addressed concerning the presentation of the material by the 
experimenter. In social we have wrestled with this issue (e.g. Barnes 
and Rosenthal). Best to you all and thanks for an insightful discussion.
Todd.

Barnes, M. L. & Rosenthal, R. (1985). Interpersonal effects of 
experimenter attractiveness, attire, and gender. Journal of Personality 
and Social Psychology, 48, 435-446.


Eran Magen wrote:

>I am absolutely loving this thread, and am so glad to see this topic
>discussed openly and earnestly. The problem of failed replications has
>a strong negative effect on the field, since the experimental
>community seems to "canonize" published results, and consider any
>failed replications to be faulty - even if results were reported just
>once, or only by the same author time and again. There are a number of
>well-known and oft-cited experiments which seem to never be replicated
>outside of the original publishing group. Reporting results that
>cannot be reproduced (perhaps because the original outcome happened by
>chance alone, or because the methods are under-specified) is what
>prevents psychology from becoming a truly incremental science.
>
>In the spirit of brainstorming possible solution, here's an idea that
>I've been discussing with friends and colleagues over the past year or
>so, as this topic has been on my mind.
>
>1) As a requirement for publication, certain journals would require a
>"seal of replicability" (the process for receiving one is described
>below). I imagine a journal that only published studies which have
>been proven to be replicable by a third party would garner a *lot* of
>attention and regard.
>2) Upon receiving a submission and following inital triage, the editor
>would send the manuscript and any accompanying materials to a central
>replication center (which can be coordinated by the APA, NSF, NIH...
>or multiple agencies could exist for different fields of study).
>3) The center would either carry out the replication study by itself
>(using in-house researchers and facilities) or outsource this task
>(e.g. by sending it to participating MA programs, as suggested earlier
>in this thread). The study will be designed by following the "methods"
>section of the manuscript (and any supporting online materials), and
>using any accompanying material as stimuli.
>4) Successful replications would be reported back to the journal (via
>the coordinating center), which would then proceed with the
>publication process.
>
>
>* A few obvious objections to this method:
>
>1) This method won't work for all types of studies. What if an
>experiment used very uncommon equipment (e.g. life-sized elephant
>robots) or very expensive procedures (e.g. fMRI)? This would require
>too much effort to replicate.
>
>- True. We can start by replicating studies that *are* easy to
>conduct, such as the common stare-and-click studies, or others that do
>not require expensive procedures or materials. If there is an unusual
>set of stimuli involved in the process, the author can send those in
>(well, perhaps not life-sized elephant robot, but definitely any
>unusual visual stimuli, audio files, etc.).
>
>2) The publication process will be slowed down.
>
>- True, it would. However, not by much, if the author provides all of
>the necessary stimuli and a clear methods section. The replication
>centers, after all, will be churning those things out, and so will
>have lots of participants and dedicated researchers standing by. This
>can even happen as the reviewers go over the paper, to save time.
>
>3) Operating these replication centers will be expensive, and nobody
>will give the money.
>
>- We are losing much more (money and credibility) by NOT having such
>replication centers. Tremendous amounts of resources are given to
>researchers who report sexy-yet-irreplicable studies, and the entire
>discipline (and, to be just a little idealistic, humanity at large) is
>constantly paying the price for supporting a non-incremental science.
>Implementing a system of this sort will guarantee a much more
>thoughtful, careful, and robust system of experimentation.
>
>4) What if the replication center will be using a different population
>than the original experimenter?
>
>- It's up to the experimenter to specify whether or not she believes
>that the results are generalizable, or whether they only pertain to
>specific parts of the population. There's no reason to believe that a
>replication center (or any of its collaborators) will have any trouble
>with finding participants. Again, this can only lead to more careful
>claims, and more thoughtful discussions of results.
>
>5) What if lots of studies won't be replicated?
>
>- Good! This will push us to take ourselves seriously, and take steps
>to become a truly incremental science. Perhaps we'll need to change
>criteria for significance. We'll probably need to start being very
>specific about our methods, which will naturally lead to less wild
>claims about their general applicability.
>
>If You've read this Email this far - hat's off, and thanks for taking
>the time. :-) I'm glad (as usual) to be a part of this wonderful JDM
>group, which is willing to take on an issue as delicate and as crucial
>as this. I believe good things will come out of this discussion.
>
>Take care, enjoy the meanwhile,
>
>- Eran Magen
>
>
>@-`-,-'--
>
>Have You visited http://www.TheHungerSite.com today? You really get to
>send food to hungry people, and it really only costs You 5 seconds and
>a click.
>
>
>On 5/6/07, mdoher2 at bgnet.bgsu.edu <mdoher2 at bgnet.bgsu.edu> wrote:
>  
>
>>My first week as a faculty member at Bowling Green was in September of 1965. The Ohio
>>Board of Regents had approved our PhD program in August. That first week was a week-long
>>faculty meeting, dedicated almost exclusively to putting the meat onto the bones of the
>>approved doctoral proposal. For a few years after that, I kept promoting the idea that an
>>acceptable Master's thesis should be a literal replication, to the extent permitted by the
>>quality of the Method section of the paper that had reported the experiment being replicated.
>>To the best of my recollection, not a single person agreed! I still think it is a good idea, and it
>>would not create another requirement (I comletely concur with Jon's terror of adding yet
>>another requirement) and it would likely get the students into an active research mode much
>>sooner.
>>
>>There is a problem. A couple of years later, I taught a seminar in cognitive, and had as a
>>requirement that the students select an important study that could be replicated with the
>>resources at hand, but they only had to run a single subject. The purposes of doing so were
>>to get them to read more closely than they ever had before, and to get them to see the
>>imprtance of writing clear and complete method sections. Every student concluded that they
>>could not do a truly literal replication without contacting the author(s) of the paper!
>>
>>Perhaps stressing the need for replication might have a side effect of more complete
>>writeups?
>>
>>Mike Doherty
>>
>>
>>_______________________________________________
>>Jdm-society mailing list
>>Jdm-society at mail.sjdm.org
>>http://www.sjdm.org/mailman/listinfo/jdm-society
>>    
>>
>_______________________________________________
>Jdm-society mailing list
>Jdm-society at mail.sjdm.org
>http://www.sjdm.org/mailman/listinfo/jdm-society
>  
>


-- 
Todd McElroy Ph.D.
Psychology Department
Appalachian State University
Boone, N.C.  28608
(828) 262-2720 office
(828) 262-2974 fax

Try to enjoy life as it passes by.