[Jdm-society] On Replication

[prestos at umich.edu]

Addressing the prior comment about cost/benefit in replicating studies, 
it seems that the most parsimonious solution to address all concerns is 
this:

Create *one* psychology journal for Null Results (not one per 
subfield). This journal will be like the JEP:General of unreplicable 
results (i.e., thorough and important). Thus, papers submitted to this 
journal will consist of multiple-experiment investigations into a prior 
effect where the authors tried, in total earnestness, to replicate the 
effect. In one experiment, the original method will be used with a very 
large sample size, in other experiments, the stimuli will be altered to 
maximize effect sizes, in most cases, the original author will be 
contacted to trouble-shoot the methods (boundary conditions become part 
of the story) - in short, every attempt will be made to get the 
experiment to work.

This solves many problems: 1) No new requirements. 2) Because you can 
publish the results, you subvert the CV-damaging effects of solutions 
that take the form of "exercises." 3) Now when you try to use a 
paradigm from another lab but fail three or four times to get it to 
work, you have a possible exciting publication instead of a lost year 
of your life. 4) Because there is only one journal, there is 
competition to get your paper in there, ensuring that replications are 
not published that were not honest attempts, or suffered for ridiculous 
reasons like Power. 5) Because there is competition for publication, 
the best bet will be to try to replicate famous studies that are 
cornerstones of Psychology, but are rumored to not replicate (I can 
think of three examples off the bat). 6) Because of the aforementioned 
reasons, we are sure to get exciting and real results out of the 
process (perhaps publishing on an as-needed basis to further ensure 
quality).

This would greatly add to the integrity of the field because 1) we 
would not be basing the premises of entire fields on false beliefs and 
2) people would be more hesitant to publish papers of dubious origin in 
the service of great fame (these would attract even *more* attempts at 
replication, thus bringing great humiliation).

Most Sincerely,
Stephanie D. Preston

Assistant Professor
Cognition and Perception Area
University of Michigan
prestos at umich.edu
http://www.itd.umich.edu/~prestos/