[Jdm-society] On replications

[Dr. George Quattrone]

My fellow jdm socialites;

The morning before last, I awoke up from what I foolishly took to be a
dream.
Having learned this week about the necessity of replication for establishing
scientific truth, I had no choice but to conclude that what I took to be a
dream must have been something else .
It had to be something else because, the following night, I returned to bed
and reproduced all the situational and attitudinal factors that prevailed
when I went to bed the night before.
Yet I did not have the same dream. This time I dreamt about , ... oh, never
mind. In brief, my attempt to replicate the initial conditions to reproduce
the same dreamy response led to data that, according to all measures of
proximity that I have heard of, from Multidimensional Scaling to Similarity
Trees, were so remote from statistical significance I would report the
results with a letter other than "p" and a symbol other than "<".

I failed to reproduce a dream, under conditions I could not imagine to be
any more identical to the original conditions that first produced that
dream.
The widely held belief that people have things called "dreams" at night, or
whenever they decide to snooze, must be wrong.

In case the above is unclear, here it is in plain talk:

The experience that followed my going nighty-night last night was so
discrepant from the experience that my going nighty-night tonight, in spite
of al measures I took to make the experience as identical as possible, leads
me to conclude that dreams are mythical psychological concepts. If they were
not mythical, why could you not replicate them! After all, replication, it
has been proven in this thread, is the true hallmark of SCIENCE.
George

-----Original Message-----
From: jdm-society-bounces at mail.sjdm.org
[mailto:jdm-society-bounces at mail.sjdm.org]On Behalf Of
jdm-society-request at mail.sjdm.org
Sent: Sunday, May 06, 2007 9:16 PM
To: jdm-society at mail.sjdm.org
Subject: Jdm-society Digest, Vol 101, Issue 10

Send Jdm-society mailing list submissions to
        jdm-society at mail.sjdm.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://www.sjdm.org/mailman/listinfo/jdm-society
or, via email, send a message with subject or body 'help' to
        jdm-society-request at mail.sjdm.org

You can reach the person managing the list at
        jdm-society-owner at mail.sjdm.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Jdm-society digest..."


Today's Topics:

   1. Re: Replications (mdoher2 at bgnet.bgsu.edu)
   2. Re: Replications (acr.martins)
   3. Re: Replications (Eran Magen)
   4. Re: Replications (Todd McElroy)


----------------------------------------------------------------------

Message: 1
Date: Sun,  6 May 2007 17:53:06 -0400
From: "mdoher2 at bgnet.bgsu.edu" <mdoher2 at bgnet.bgsu.edu>
Subject: Re: [Jdm-society] Replications
To: "Jonathan Baron" <baron at psych.upenn.edu>
Cc: JDM-Society <jdm-society at mail.sjdm.org>
Message-ID: <1178488386-28888.00023.00571-smmsdV2.1.6 at smtp.bgsu.edu>

My first week as a faculty member at Bowling Green was in September of 1965.
The Ohio
Board of Regents had approved our PhD program in August. That first week was
a week-long
faculty meeting, dedicated almost exclusively to putting the meat onto the
bones of the
approved doctoral proposal. For a few years after that, I kept promoting the
idea that an
acceptable Master's thesis should be a literal replication, to the extent
permitted by the
quality of the Method section of the paper that had reported the experiment
being replicated.
To the best of my recollection, not a single person agreed! I still think it
is a good idea, and it
would not create another requirement (I comletely concur with Jon's terror
of adding yet
another requirement) and it would likely get the students into an active
research mode much
sooner.

There is a problem. A couple of years later, I taught a seminar in
cognitive, and had as a
requirement that the students select an important study that could be
replicated with the
resources at hand, but they only had to run a single subject. The purposes
of doing so were
to get them to read more closely than they ever had before, and to get them
to see the
imprtance of writing clear and complete method sections. Every student
concluded that they
could not do a truly literal replication without contacting the author(s) of
the paper!

Perhaps stressing the need for replication might have a side effect of more
complete
writeups?

Mike Doherty




------------------------------

Message: 2
Date: Sun,  6 May 2007 23:12:37 -0300
From: "acr.martins" <acr.martins at uol.com.br>
Subject: Re: [Jdm-society] Replications
To: "mdoher2" <mdoher2 at bgnet.bgsu.edu>
Cc: jdm-society <jdm-society at mail.sjdm.org>
Message-ID: <JHNFH1$20BE9A98193D4A67307A6066DEB185F4 at uol.com.br>
Content-Type: text/plain; charset=iso-8859-1

I must say that I completely agree with Mike Doherty, at least about
dissertations in experimental areas. Something similar might be difficult to
implement in purely theoretical subjects (I am thinking of things like
theoretical Physics or, even worse, Mathematics), where simply making the
same calculations presented in an old paper is a task too easy, once the
original author has paved the way. That is, unless the student finds a new
way of proving the same thing. And that would be, quite often, asking too
much. For everyone else, the idea that every Master's dissertation should be
a replication seems perfect and it would make the results in every area far
more reliable.

And it is really necessary! On the reliability of the reported results as a
whole, when one conducts no experience, I have made a few simple but
disturbing calculations myself, by assuming a Bayesian reader who tries to
decide between two theories. The reader does not perform experiments, only
read descriptions of experiments done by others. If the reader includes in
the analysis the possibility of error and deception, under some
circumstances, no certainty is ever reached, even in the limit of infinite
articles read.

Victor Palmer has extended that result for the case where the reader only
uses replicated results as his basis. That seemed to solve the problem, but
his analysis is only valid under some special circumstances. Still, it makes
it very clear the need for replication.

If anyone is interested in those results, the references are

Martins, Andr? C. R. (2005). Deception and Convergence of Opinions.
JASSS-The Journal of Artificial Societies and Social Simulation, v. 8, n. 2.

Palmer, Victor (2006) Deception and Convergence of Opinions Part 2: the
Effects of Reproducibility. JASSS-The Journal of Artificial Societies and
Social Simulation, v. 9, n. 1.


Andr? C. R. Martins
Escola de Artes, Ci?ncias e Humanidades
Universidade de S?o Paulo
Brazil


> My first week as a faculty member at Bowling Green was in September of
1965. The Ohio
> Board of Regents had approved our PhD program in August. That first week
was a week-long
> faculty meeting, dedicated almost exclusively to putting the meat onto the
bones of the
> approved doctoral proposal. For a few years after that, I kept promoting
the idea that an
> acceptable Master's thesis should be a literal replication, to the extent
permitted by the
> quality of the Method section of the paper that had reported the
experiment being replicated.
> To the best of my recollection, not a single person agreed! I still think
it is a good idea, and it
> would not create another requirement (I comletely concur with Jon's terror
of adding yet
> another requirement) and it would likely get the students into an active
research mode much
> sooner.
>
> There is a problem. A couple of years later, I taught a seminar in
cognitive, and had as a
> requirement that the students select an important study that could be
replicated with the
> resources at hand, but they only had to run a single subject. The purposes
of doing so were
> to get them to read more closely than they ever had before, and to get
them to see the
> imprtance of writing clear and complete method sections. Every student
concluded that they
> could not do a truly literal replication without contacting the author(s)
of the paper!
>
> Perhaps stressing the need for replication might have a side effect of
more complete
> writeups?
>
> Mike Doherty




------------------------------

Message: 3
Date: Sun, 6 May 2007 19:53:27 -0700
From: "Eran Magen" <emaileranmagen at gmail.com>
Subject: Re: [Jdm-society] Replications
To: JDM-Society <jdm-society at mail.sjdm.org>
Cc: SPL <psyphylab at lists.stanford.edu>
Message-ID:
        <94632a370705061953j1b921e42l8041e633b3baabab at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

I am absolutely loving this thread, and am so glad to see this topic
discussed openly and earnestly. The problem of failed replications has
a strong negative effect on the field, since the experimental
community seems to "canonize" published results, and consider any
failed replications to be faulty - even if results were reported just
once, or only by the same author time and again. There are a number of
well-known and oft-cited experiments which seem to never be replicated
outside of the original publishing group. Reporting results that
cannot be reproduced (perhaps because the original outcome happened by
chance alone, or because the methods are under-specified) is what
prevents psychology from becoming a truly incremental science.

In the spirit of brainstorming possible solution, here's an idea that
I've been discussing with friends and colleagues over the past year or
so, as this topic has been on my mind.

1) As a requirement for publication, certain journals would require a
"seal of replicability" (the process for receiving one is described
below). I imagine a journal that only published studies which have
been proven to be replicable by a third party would garner a *lot* of
attention and regard.
2) Upon receiving a submission and following inital triage, the editor
would send the manuscript and any accompanying materials to a central
replication center (which can be coordinated by the APA, NSF, NIH...
or multiple agencies could exist for different fields of study).
3) The center would either carry out the replication study by itself
(using in-house researchers and facilities) or outsource this task
(e.g. by sending it to participating MA programs, as suggested earlier
in this thread). The study will be designed by following the "methods"
section of the manuscript (and any supporting online materials), and
using any accompanying material as stimuli.
4) Successful replications would be reported back to the journal (via
the coordinating center), which would then proceed with the
publication process.


* A few obvious objections to this method:

1) This method won't work for all types of studies. What if an
experiment used very uncommon equipment (e.g. life-sized elephant
robots) or very expensive procedures (e.g. fMRI)? This would require
too much effort to replicate.

- True. We can start by replicating studies that *are* easy to
conduct, such as the common stare-and-click studies, or others that do
not require expensive procedures or materials. If there is an unusual
set of stimuli involved in the process, the author can send those in
(well, perhaps not life-sized elephant robot, but definitely any
unusual visual stimuli, audio files, etc.).

2) The publication process will be slowed down.

- True, it would. However, not by much, if the author provides all of
the necessary stimuli and a clear methods section. The replication
centers, after all, will be churning those things out, and so will
have lots of participants and dedicated researchers standing by. This
can even happen as the reviewers go over the paper, to save time.

3) Operating these replication centers will be expensive, and nobody
will give the money.

- We are losing much more (money and credibility) by NOT having such
replication centers. Tremendous amounts of resources are given to
researchers who report sexy-yet-irreplicable studies, and the entire
discipline (and, to be just a little idealistic, humanity at large) is
constantly paying the price for supporting a non-incremental science.
Implementing a system of this sort will guarantee a much more
thoughtful, careful, and robust system of experimentation.

4) What if the replication center will be using a different population
than the original experimenter?

- It's up to the experimenter to specify whether or not she believes
that the results are generalizable, or whether they only pertain to
specific parts of the population. There's no reason to believe that a
replication center (or any of its collaborators) will have any trouble
with finding participants. Again, this can only lead to more careful
claims, and more thoughtful discussions of results.

5) What if lots of studies won't be replicated?

- Good! This will push us to take ourselves seriously, and take steps
to become a truly incremental science. Perhaps we'll need to change
criteria for significance. We'll probably need to start being very
specific about our methods, which will naturally lead to less wild
claims about their general applicability.

If You've read this Email this far - hat's off, and thanks for taking
the time. :-) I'm glad (as usual) to be a part of this wonderful JDM
group, which is willing to take on an issue as delicate and as crucial
as this. I believe good things will come out of this discussion.

Take care, enjoy the meanwhile,

- Eran Magen


@-`-,-'--

Have You visited http://www.TheHungerSite.com today? You really get to
send food to hungry people, and it really only costs You 5 seconds and
a click.


On 5/6/07, mdoher2 at bgnet.bgsu.edu <mdoher2 at bgnet.bgsu.edu> wrote:
> My first week as a faculty member at Bowling Green was in September of
1965. The Ohio
> Board of Regents had approved our PhD program in August. That first week
was a week-long
> faculty meeting, dedicated almost exclusively to putting the meat onto the
bones of the
> approved doctoral proposal. For a few years after that, I kept promoting
the idea that an
> acceptable Master's thesis should be a literal replication, to the extent
permitted by the
> quality of the Method section of the paper that had reported the
experiment being replicated.
> To the best of my recollection, not a single person agreed! I still think
it is a good idea, and it
> would not create another requirement (I comletely concur with Jon's terror
of adding yet
> another requirement) and it would likely get the students into an active
research mode much
> sooner.
>
> There is a problem. A couple of years later, I taught a seminar in
cognitive, and had as a
> requirement that the students select an important study that could be
replicated with the
> resources at hand, but they only had to run a single subject. The purposes
of doing so were
> to get them to read more closely than they ever had before, and to get
them to see the
> imprtance of writing clear and complete method sections. Every student
concluded that they
> could not do a truly literal replication without contacting the author(s)
of the paper!
>
> Perhaps stressing the need for replication might have a side effect of
more complete
> writeups?
>
> Mike Doherty
>
>
> _______________________________________________
> Jdm-society mailing list
> Jdm-society at mail.sjdm.org
> http://www.sjdm.org/mailman/listinfo/jdm-society


------------------------------

Message: 4
Date: Mon, 07 May 2007 00:15:11 -0400
From: Todd McElroy <mcelroygt at appstate.edu>
Subject: Re: [Jdm-society] Replications
Cc: JDM-Society <jdm-society at mail.sjdm.org>
Message-ID: <463EA7CF.4000004 at appstate.edu>
Content-Type: text/plain; charset=windows-1252; format=flowed


I agree completely with this point and I believe that it would greatly
improve the validity (and reliability) of our science. However, I also
agree with the logistical issues that you rightly point out. I would
suggest that this process might be more likely if it were to be
introduced incrementally. Specifically, this could be presented as an
option to authors or prospective authors. During the review process the
stimuli could be sent out for ??review?? as well. At this point you should
ask why someone would want to take such a risk? I would suggest that a
replication of the study presented in a paper would certainly seem to
make the paper more attractive to an editor who might be in limbo about
the work. I would also like to offer another important issue that should
be addressed concerning the presentation of the material by the
experimenter. In social we have wrestled with this issue (e.g. Barnes
and Rosenthal). Best to you all and thanks for an insightful discussion.
Todd.

Barnes, M. L. & Rosenthal, R. (1985). Interpersonal effects of
experimenter attractiveness, attire, and gender. Journal of Personality
and Social Psychology, 48, 435-446.


Eran Magen wrote:

>I am absolutely loving this thread, and am so glad to see this topic
>discussed openly and earnestly. The problem of failed replications has
>a strong negative effect on the field, since the experimental
>community seems to "canonize" published results, and consider any
>failed replications to be faulty - even if results were reported just
>once, or only by the same author time and again. There are a number of
>well-known and oft-cited experiments which seem to never be replicated
>outside of the original publishing group. Reporting results that
>cannot be reproduced (perhaps because the original outcome happened by
>chance alone, or because the methods are under-specified) is what
>prevents psychology from becoming a truly incremental science.
>
>In the spirit of brainstorming possible solution, here's an idea that
>I've been discussing with friends and colleagues over the past year or
>so, as this topic has been on my mind.
>
>1) As a requirement for publication, certain journals would require a
>"seal of replicability" (the process for receiving one is described
>below). I imagine a journal that only published studies which have
>been proven to be replicable by a third party would garner a *lot* of
>attention and regard.
>2) Upon receiving a submission and following inital triage, the editor
>would send the manuscript and any accompanying materials to a central
>replication center (which can be coordinated by the APA, NSF, NIH...
>or multiple agencies could exist for different fields of study).
>3) The center would either carry out the replication study by itself
>(using in-house researchers and facilities) or outsource this task
>(e.g. by sending it to participating MA programs, as suggested earlier
>in this thread). The study will be designed by following the "methods"
>section of the manuscript (and any supporting online materials), and
>using any accompanying material as stimuli.
>4) Successful replications would be reported back to the journal (via
>the coordinating center), which would then proceed with the
>publication process.
>
>
>* A few obvious objections to this method:
>
>1) This method won't work for all types of studies. What if an
>experiment used very uncommon equipment (e.g. life-sized elephant
>robots) or very expensive procedures (e.g. fMRI)? This would require
>too much effort to replicate.
>
>- True. We can start by replicating studies that *are* easy to
>conduct, such as the common stare-and-click studies, or others that do
>not require expensive procedures or materials. If there is an unusual
>set of stimuli involved in the process, the author can send those in
>(well, perhaps not life-sized elephant robot, but definitely any
>unusual visual stimuli, audio files, etc.).
>
>2) The publication process will be slowed down.
>
>- True, it would. However, not by much, if the author provides all of
>the necessary stimuli and a clear methods section. The replication
>centers, after all, will be churning those things out, and so will
>have lots of participants and dedicated researchers standing by. This
>can even happen as the reviewers go over the paper, to save time.
>
>3) Operating these replication centers will be expensive, and nobody
>will give the money.
>
>- We are losing much more (money and credibility) by NOT having such
>replication centers. Tremendous amounts of resources are given to
>researchers who report sexy-yet-irreplicable studies, and the entire
>discipline (and, to be just a little idealistic, humanity at large) is
>constantly paying the price for supporting a non-incremental science.
>Implementing a system of this sort will guarantee a much more
>thoughtful, careful, and robust system of experimentation.
>
>4) What if the replication center will be using a different population
>than the original experimenter?
>
>- It's up to the experimenter to specify whether or not she believes
>that the results are generalizable, or whether they only pertain to
>specific parts of the population. There's no reason to believe that a
>replication center (or any of its collaborators) will have any trouble
>with finding participants. Again, this can only lead to more careful
>claims, and more thoughtful discussions of results.
>
>5) What if lots of studies won't be replicated?
>
>- Good! This will push us to take ourselves seriously, and take steps
>to become a truly incremental science. Perhaps we'll need to change
>criteria for significance. We'll probably need to start being very
>specific about our methods, which will naturally lead to less wild
>claims about their general applicability.
>
>If You've read this Email this far - hat's off, and thanks for taking
>the time. :-) I'm glad (as usual) to be a part of this wonderful JDM
>group, which is willing to take on an issue as delicate and as crucial
>as this. I believe good things will come out of this discussion.
>
>Take care, enjoy the meanwhile,
>
>- Eran Magen
>
>
>@-`-,-'--
>
>Have You visited http://www.TheHungerSite.com today? You really get to
>send food to hungry people, and it really only costs You 5 seconds and
>a click.
>
>
>On 5/6/07, mdoher2 at bgnet.bgsu.edu <mdoher2 at bgnet.bgsu.edu> wrote:
>
>
>>My first week as a faculty member at Bowling Green was in September of
1965. The Ohio
>>Board of Regents had approved our PhD program in August. That first week
was a week-long
>>faculty meeting, dedicated almost exclusively to putting the meat onto the
bones of the
>>approved doctoral proposal. For a few years after that, I kept promoting
the idea that an
>>acceptable Master's thesis should be a literal replication, to the extent
permitted by the
>>quality of the Method section of the paper that had reported the
experiment being replicated.
>>To the best of my recollection, not a single person agreed! I still think
it is a good idea, and it
>>would not create another requirement (I comletely concur with Jon's terror
of adding yet
>>another requirement) and it would likely get the students into an active
research mode much
>>sooner.
>>
>>There is a problem. A couple of years later, I taught a seminar in
cognitive, and had as a
>>requirement that the students select an important study that could be
replicated with the
>>resources at hand, but they only had to run a single subject. The purposes
of doing so were
>>to get them to read more closely than they ever had before, and to get
them to see the
>>imprtance of writing clear and complete method sections. Every student
concluded that they
>>could not do a truly literal replication without contacting the author(s)
of the paper!
>>
>>Perhaps stressing the need for replication might have a side effect of
more complete
>>writeups?
>>
>>Mike Doherty
>>
>>
>>_______________________________________________
>>Jdm-society mailing list
>>Jdm-society at mail.sjdm.org
>>http://www.sjdm.org/mailman/listinfo/jdm-society
>>
>>
>_______________________________________________
>Jdm-society mailing list
>Jdm-society at mail.sjdm.org
>http://www.sjdm.org/mailman/listinfo/jdm-society
>
>


--
Todd McElroy Ph.D.
Psychology Department
Appalachian State University
Boone, N.C.  28608
(828) 262-2720 office
(828) 262-2974 fax

Try to enjoy life as it passes by.



------------------------------

_______________________________________________
Jdm-society mailing list
Jdm-society at mail.sjdm.org
http://www.sjdm.org/mailman/listinfo/jdm-society


End of Jdm-society Digest, Vol 101, Issue 10
********************************************