[Jdm-society] On the need for replications

[Bertram F. Malle]

Hi Scott,

I couldn't agree with you more about the alarming absence of  
replication studies but, I would argue, not just in marketing  
research but in the field of behavioral science more broadly.   Even  
worse, there may actually be failed replications lurking in the  
(massive) literature that people haven't noticed. Just last year I  
came out with a meta-analysis of all published articles on the  
classic actor-observer asymmetry in causal attribution (not even  
looking into file drawers), and it showed  that there is virtually no  
evidence for this asymmetry.  For thirty years, every textbook in  
social psychology has taken this hypothesis to be robustly and  
strongly supported; but it isn't.

One caveat on replication percentages, however. Because of the overly  
stringent alpha levels we tend to use in our disciplines and because  
of the journals' refusal to publish nonsignificant findings,  
replication studies of previously published studies have a low chance  
of replicating for statistical reasons alone.  Schmidt (1996) showed  
that replications can be expected to to succeed in only about 40% of  
the cases even if the effect is there.  That is because, to be  
significant at a stringent alpha (and typically low power to begin  
with), many original studies had to have had a larger observed effect  
size than the true population effect is, and regression to the mean  
makes it likely that the next study shows a weaker effect size and  
may not be significant (again, because we tend to have underpowered  
studies).   So if we start to increase replication attempts, we also  
need to start weighting the importance of statistical power more and  
the importance of alpha error less.

Here are the two references, for your interest.

Malle, B. F. (2006b). The actor-observer asymmetry in causal  
attribution: A (surprising) meta-analysis. Psychological Bulletin,  
132, 895-919.

Schmidt, F. (1996). Statistical significance testing and cumulative  
knowledge in psychology: Implications for training of researchers.   
Psychological Methods, 1, 115-129.

Best,

Bertram



On May 4, 2007, at 7:02 PM, armstrong wrote:

> Scientists regard replications as the bedrock of research.  
> Researchers in marketing have expressed concerns over what seems to  
> be a paucity of replications. Prior research also suggests that the  
> situation in marketing is similar to that in other areas of  
> management science.
>
> This concern is important because a large percentage of  
> replications have failed to support the original findings. Our  
> study shows that only between 25% and 44% of replication studies  
> confirmed earlier results - a figure that questions the findings in  
> marketing that come from one-shot-studies.
>
> In line with these observations, editorial policies of some leading  
> marketing journals have been modified to encourage more  
> replications. We conducted an extension of a 1994 study see whether  
> these efforts have had an effect. In fact, the replication rate has  
> fallen to 1.2 percent, a decrease in the rate of about half. As  
> things now stand, practitioners should be skeptical about using the  
> results published in marketing journals as hardly any of them have  
> been successfully replicated, teachers are advised to ignore  
> findings until they have been replicated, and researchers should  
> put little stock in the outcomes of one-shot studies.
>
> This paper, "Replication research's disturbing trend," by  
> Evanschitzky, H., C. Baumgarth, R. Hubbard & J. S. Armstrong,  
> Journal of Business Research, 60 (2007), 411-415, is available in  
> full-text at  http://jscottarmstrong.com
>
> Scott Armstrong
>
>
> -- 
> J. Scott Armstrong
> The Wharton School, U. of PA, Phila, PA 19104
> http://www.jscottarmstrong.com
>
> _______________________________________________
> Jdm-society mailing list
> Jdm-society at mail.sjdm.org
> http://www.sjdm.org/mailman/listinfo/jdm-society


Bertram F. Malle, Ph.D.
Associate Professor of Psychology
Director, Institute of Cognitive and Decision Sciences
1227 University of Oregon
Eugene OR 97403-1227

Phone: (541) 346-0475
Fax: (541) 346-4911
Web: www.uoregon.edu/~bfmalle/



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.sjdm.org/mail-archive/jdm-society/attachments/20070504/26dbff9c/attachment-0001.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: unknown.gif
Type: image/gif
Size: 1241 bytes
Desc: not available
Url : http://www.sjdm.org/mail-archive/jdm-society/attachments/20070504/26dbff9c/attachment-0001.gif