[Jdm-society] Replications

[David J. Weiss]

David Budesco has hit the nail on the head. Who wants to be known as a
"mere" replicator?  In fact, for years Jim Shanteau and I have given
unofficial Xerox awards to speakers whose studies seem to lack originality.

A replication study will generally be considered "interesting" enough to
publish on its own only if it yields a disconfirming result. Even that might
not be enough; the power issue has been discussed. One needs a story about 
why the replication was expected to fail, a story that exposes the flaws in 
the logic or procedure of the original study. Similarly, granting agencies 
will not fund a replication study unless the requestor has a great
explanation of why the replication ought to fail. These realities mean that
even the few replication studies that are currently attempted are carried
out by researchers looking to denigrate the original results.

The only practical way to change things is by creating a neutral incentive
to do a replication study. The easiest way to accomplish this would be to
embed a replication requirement into the Ph.D. program. Every student would 
be asked to replicate an important study as part of the first or second year 
requirement. The original study would have to have been carried out at a 
different institution, and by a research team having no conflict of interest 
with the student's mentor. Whether the results come out "right" or "wrong" 
should have no bearing on whether the requirement has been satisfied. After 
the student's mentor approved the paper, it would be submitted for 
publication. Where, you ask?

While Jon might be willing to allocate a portion of the SJDM journal space
for such papers, there would be far too many to put there, so the solution
would be to create an independent Journal of Replication Studies. Each field 
would need its own. We would have the Journal of JDM Replication Studies. 
The journals would have to be electronic, as they will not be profitable. 
Few scholars would read the Journal regularly. But so long as the papers 
were indexed and accessible, those who cared about a particular topic could 
get an accurate picture of the experimental results. Also, students choosing 
a study to replicate would find it easy to see how many times a study has 
been replicated previously.

To keep the reviewing burden managable, I would suggest that an author of 
the original study, if available, should be the sole reviewer. At least that
way, it would be possible to find a reviewer who thought the topic was
interesting. Original authors should be flattered that their study was
considered important enough to replicate. The criteria for acceptance would 
focus only on procedure and presentation, and would not include the 
direction of the results.

David

----- Original Message ----- 
From: "David Budescu" <dbudescu at cyrus.psych.uiuc.edu>
To: "JDM-Society" <jdm-society at mail.sjdm.org>
Sent: Saturday, May 05, 2007 9:03 AM
Subject: [Jdm-society] replications



 Jon should be congratulated for his editorial policy, but I still
 predict that we'll see relatively few "straight" replications (fully or
 partially successful) in press.  Part of the problem is that the incentive
 structure in our profession is biased against this.  We value innovation,
 originality and creativity and we reward them. Replications are often
 dismissed as "more of the same" or "lacking any new ideas".

 Besides the obvious element of self-selection (recall why most of us got
 into this business), this attitide permeates our institutions.
 This may not be always stated explicitly, but it is implicit in
 choices made under limited resources (given a fixed amount of grant
 money to distribute, or a limited number of journal pages to fill,
 granting agencies and journal editors favor new "exciting" ideas, even if 
they recognize, and pay lip service, to the importance of replications).
 And, think of hiring and promotion committees...

 David