[Jdm-society] On the need for replications

[Robyn Dawes]

    ³Getting  it right² implies a good pr. of replication, sampling from the
same population.    That is easily estimated. The real replication  problem
IN OUR AREAS arises if we change the population or sampling in a way that
³shouldn¹t matter,² only it does.

Robyn


On 5/5/07 11:20 AM, "armstrong" <armstrong at wharton.upenn.edu> wrote:

> Thanks, for your message, Bertram.
> 
> You are correct about the statistical significance issue. Ray Hubbard and I
> had planned to reanalyze the data using only direction of effects, but we
> never got around to it.
> 
> Also, in recent years I have come to the conclusion (aided by Schmidt and
> Hunter, of course) that statistical significance has no benefits. The short
> version is J. Scott Armstrong (2007), "Statistical Significance Tests are
> Unnecessary Even When Properly Done and Properly Interpreted: Reply to
> Commentaries", International Journal of Forecasting (forthcoming)  in full
> text at http://jscottarmstrong.com and the longer is J. Scott Armstrong,
> "Significance Tests Harm Progress in Forecasting" International Journal of
> Forecasting (forthcoming) on the same website.
> 
> Scott
> 
> 
>> Hi Scott,
>> 
>> I couldn't agree with you more about the alarming absence of replication
>> studies but, I would argue, not just in marketing research but in the field
>> of behavioral science more broadly.   Even worse, there may actually be
>> failed replications lurking in the (massive) literature that people haven't
>> noticed. Just last year I came out with a meta-analysis of all published
>> articles on the classic actor-observer asymmetry in causal attribution (not
>> even looking into file drawers), and it showed  that there is virtually no
>> evidence for this asymmetry.  For thirty years, every textbook in social
>> psychology has taken this hypothesis to be robustly and strongly supported;
>> but it isn't.
>> 
>> One caveat on replication percentages, however. Because of the overly
>> stringent alpha levels we tend to use in our disciplines and because of the
>> journals' refusal to publish nonsignificant findings, replication studies of
>> previously published studies have a low chance of replicating for statistical
>> reasons alone.  Schmidt (1996) showed that replications can be expected to to
>> succeed in only about 40% of the cases even if the effect is there.  That is
>> because, to be significant at a stringent alpha (and typically low power to
>> begin with), many original studies had to have had a larger observed effect
>> size than the true population effect is, and regression to the mean makes it
>> likely that the next study shows a weaker effect size and may not be
>> significant (again, because we tend to have underpowered studies).   So if we
>> start to increase replication attempts, we also need to start weighting the
>> importance of statistical power more and the importance of alpha error less.
>> 
>> Here are the two references, for your interest.
>> 
>> Malle, B. F. (2006b). The actor-observer asymmetry in causal attribution: A
>> (surprising) meta-analysis. Psychological Bulletin, 132, 895-919.
>> 
>> Schmidt, F. (1996). Statistical significance testing and cumulative knowledge
>> in psychology: Implications for training of researchers.
>> Psychological Methods, 1, 115-129.
>> 
>> Best,
>> 
>> Bertram
>> 
>> 
>> 
>> On May 4, 2007, at 7:02 PM, armstrong wrote:
>>> Scientists regard replications as the bedrock of research. Researchers in
>>> marketing have expressed concerns over what seems to be a paucity of
>>> replications. Prior research also suggests that the situation in marketing
>>> is similar to that in other areas of management science.
>>> 
>>> This concern is important because a large percentage of replications have
>>> failed to support the original findings. Our study shows that only between
>>> 25% and 44% of replication studies confirmed earlier results - a figure that
>>> questions the findings in marketing that come from one-shot-studies.
>>> 
>>> In line with these observations, editorial policies of some leading
>>> marketing journals have been modified to encourage more replications. We
>>> conducted an extension of a 1994 study see whether these efforts have had an
>>> effect. In fact, the replication rate has fallen to 1.2 percent, a decrease
>>> in the rate of about half. As things now stand, practitioners should be
>>> skeptical about using the results published in marketing journals as hardly
>>> any of them have been successfully replicated, teachers are advised to
>>> ignore findings until they have been replicated, and researchers should put
>>> little stock in the outcomes of one-shot studies.
>>> 
>>> This paper, "Replication research's disturbing trend," by Evanschitzky, H.,
>>> C. Baumgarth, R. Hubbard & J. S. Armstrong, Journal of Business Research, 60
>>> (2007), 411-415, is available in full-text at  http://jscottarmstrong.com
>>> 
>>> Scott Armstrong
>>> 
>>> 
>>> --
>>> J. Scott Armstrong
>>> The Wharton School, U. of PA, Phila, PA 19104
>>> http://www.jscottarmstrong.com
>>> 
>>> _______________________________________________
>>> Jdm-society mailing list
>>> Jdm-society at mail.sjdm.org
>>> http://www.sjdm.org/mailman/listinfo/jdm-society
>> 
>> ?
>> Bertram F. Malle, Ph.D.
>> Associate Professor of Psychology
>> Director, Institute of Cognitive and Decision Sciences
>> 1227 University of Oregon
>> Eugene OR 97403-1227
>> 
>> Phone: (541) 3


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.sjdm.org/mail-archive/jdm-society/attachments/20070507/eda39b22/attachment-0001.htm