[Jdm-society] On the need for replications
armstrong
armstrong at wharton.upenn.edu
Sat May 5 10:20:47 CDT 2007
Thanks, for your message, Bertram.
You are correct about the statistical significance issue. Ray Hubbard
and I had planned to reanalyze the data using only direction of
effects, but we never got around to it.
Also, in recent years I have come to the conclusion (aided by Schmidt
and Hunter, of course) that statistical significance has no benefits.
The short version is J. Scott Armstrong (2007), "Statistical
Significance Tests are Unnecessary Even When Properly Done and
Properly Interpreted: Reply to Commentaries", International Journal
of Forecasting (forthcoming) in full text at
http://jscottarmstrong.com and the longer is J. Scott Armstrong,
"Significance Tests Harm Progress in Forecasting" International
Journal of Forecasting (forthcoming) on the same website.
Scott
>Hi Scott,
>
>I couldn't agree with you more about the alarming absence of
>replication studies but, I would argue, not just in marketing
>research but in the field of behavioral science more broadly. Even
>worse, there may actually be failed replications lurking in the
>(massive) literature that people haven't noticed. Just last year I
>came out with a meta-analysis of all published articles on the
>classic actor-observer asymmetry in causal attribution (not even
>looking into file drawers), and it showed that there is virtually
>no evidence for this asymmetry. For thirty years, every textbook in
>social psychology has taken this hypothesis to be robustly and
>strongly supported; but it isn't.
>
>One caveat on replication percentages, however. Because of the
>overly stringent alpha levels we tend to use in our disciplines and
>because of the journals' refusal to publish nonsignificant findings,
>replication studies of previously published studies have a low
>chance of replicating for statistical reasons alone. Schmidt (1996)
>showed that replications can be expected to to succeed in only about
>40% of the cases even if the effect is there. That is because, to
>be significant at a stringent alpha (and typically low power to
>begin with), many original studies had to have had a larger observed
>effect size than the true population effect is, and regression to
>the mean makes it likely that the next study shows a weaker effect
>size and may not be significant (again, because we tend to have
>underpowered studies). So if we start to increase replication
>attempts, we also need to start weighting the importance of
>statistical power more and the importance of alpha error less.
>
>Here are the two references, for your interest.
>
>Malle, B. F. (2006b). The actor-observer asymmetry in causal
>attribution: A (surprising) meta-analysis. Psychological Bulletin,
>132, 895-919.
>
>Schmidt, F. (1996). Statistical significance testing and cumulative
>knowledge in psychology: Implications for training of researchers.
>Psychological Methods, 1, 115-129.
>
>Best,
>
>Bertram
>
>
>
>On May 4, 2007, at 7:02 PM, armstrong wrote:
>
>>Scientists regard replications as the bedrock of research.
>>Researchers in marketing have expressed concerns over what seems to
>>be a paucity of replications. Prior research also suggests that the
>>situation in marketing is similar to that in other areas of
>>management science.
>>
>>This concern is important because a large percentage of
>>replications have failed to support the original findings. Our
>>study shows that only between 25% and 44% of replication studies
>>confirmed earlier results - a figure that questions the findings in
>>marketing that come from one-shot-studies.
>>
>>In line with these observations, editorial policies of some leading
>>marketing journals have been modified to encourage more
>>replications. We conducted an extension of a 1994 study see whether
>>these efforts have had an effect. In fact, the replication rate has
>>fallen to 1.2 percent, a decrease in the rate of about half. As
>>things now stand, practitioners should be skeptical about using the
>>results published in marketing journals as hardly any of them have
>>been successfully replicated, teachers are advised to ignore
>>findings until they have been replicated, and researchers should
>>put little stock in the outcomes of one-shot studies.
>>
>>This paper, "Replication research's disturbing trend," by
>>Evanschitzky, H., C. Baumgarth, R. Hubbard & J. S. Armstrong,
>>Journal of Business Research, 60 (2007), 411-415, is available in
>>full-text at http://jscottarmstrong.com
>>
>>Scott Armstrong
>>
>>
>>--
>>J. Scott Armstrong
>>The Wharton School, U. of PA, Phila, PA 19104
>>http://www.jscottarmstrong.com
>>
>>_______________________________________________
>>Jdm-society mailing list
>>Jdm-society at mail.sjdm.org
>>http://www.sjdm.org/mailman/listinfo/jdm-society
>
>?
>Bertram F. Malle, Ph.D.
>Associate Professor of Psychology
>Director, Institute of Cognitive and Decision Sciences
>1227 University of Oregon
>Eugene OR 97403-1227
>
>Phone: (541) 346-0475
>Fax: (541) 346-4911
>Web: www.uoregon.edu/~bfmalle/
>
>
>
>Hi Scott,
>
>I couldn't agree with you more about the alarming absence of
>replication studies but, I would argue, not just in marketing
>research but in the field of behavioral science more broadly. Even
>worse, there may actually be failed replications lurking in the
>(massive) literature that people haven't noticed. Just last year I
>came out with a meta-analysis of all published articles on the
>classic actor-observer asymmetry in causal attribution (not even
>looking into file drawers), and it showed that there is virtually
>no evidence for this asymmetry. For thirty years, every textbook in
>social psychology has taken this hypothesis to be robustly and
>strongly supported; but it isn't.
>
>One caveat on replication percentages, however. Because of the
>overly stringent alpha levels we tend to use in our disciplines and
>because of the journals' refusal to publish nonsignificant findings,
>replication studies of previously published studies have a low
>chance of replicating for statistical reasons alone. Schmidt (1996)
>showed that replications can be expected to to succeed in only about
>40% of the cases even if the effect is there. That is because, to
>be significant at a stringent alpha (and typically low power to
>begin with), many original studies had to have had a larger observed
>effect size than the true population effect is, and regression to
>the mean makes it likely that the next study shows a weaker effect
>size and may not be significant (again, because we tend to have
>underpowered studies). So if we start to increase replication
>attempts, we also need to start weighting the importance of
>statistical power more and the importance of alpha error less.
>
>Here are the two references, for your interest.
>
>Malle, B. F. (2006b). The actor-observer asymmetry in causal
>attribution: A (surprising) meta-analysis. Psychological Bulletin,
>132, 895-919.
>
>Schmidt, F. (1996). Statistical significance testing and cumulative
>knowledge in psychology: Implications for training of researchers.
>Psychological Methods, 1, 115-129.
>
>Best,
>
>Bertram
>
>
>
>On May 4, 2007, at 7:02 PM, armstrong wrote:
>
>>Scientists regard replications as the bedrock of research.
>>Researchers in marketing have expressed concerns over what seems to
>>be a paucity of replications. Prior research also suggests that the
>>situation in marketing is similar to that in other areas of
>>management science.
>>
>>This concern is important because a large percentage of
>>replications have failed to support the original findings. Our
>>study shows that only between 25% and 44% of replication studies
>>confirmed earlier results - a figure that questions the findings in
>>marketing that come from one-shot-studies.
>>
>>In line with these observations, editorial policies of some leading
>>marketing journals have been modified to encourage more
>>replications. We conducted an extension of a 1994 study see whether
>>these efforts have had an effect. In fact, the replication rate has
>>fallen to 1.2 percent, a decrease in the rate of about half. As
>>things now stand, practitioners should be skeptical about using the
>>results published in marketing journals as hardly any of them have
>>been successfully replicated, teachers are advised to ignore
>>findings until they have been replicated, and researchers should
>>put little stock in the outcomes of one-shot studies.
>>
>>This paper, "Replication research's disturbing trend," by
>>Evanschitzky, H., C. Baumgarth, R. Hubbard & J. S. Armstrong,
>>Journal of Business Research, 60 (2007), 411-415, is available in
>>full-text at <http://jscottarmstrong.com>http://jscottarmstrong.com
>>
>>Scott Armstrong
>>
>>
>>--
>>J. Scott Armstrong
>>The Wharton School, U. of PA, Phila, PA 19104
>><http://www.jscottarmstrong.com>http://www.jscottarmstrong.com
>>
>>_______________________________________________
>>Jdm-society mailing list
>><mailto:Jdm-society at mail.sjdm.org>Jdm-society at mail.sjdm.org
>><http://www.sjdm.org/mailman/listinfo/jdm-society>http://www.sjdm.org/mailman/listinfo/jdm-society
>>
>
>
>Bertram F. Malle, Ph.D.
>Associate Professor of Psychology
>Director, Institute of Cognitive and Decision Sciences
>1227 University of Oregon
>Eugene OR 97403-1227
>
>Phone: (541) 346-0475
>Fax: (541) 346-4911
>Web: <http://www.uoregon.edu/~bfmalle/>www.uoregon.edu/~bfmalle/
--
J. Scott Armstrong
Professor of Marketing, 747 Huntsman, The Wharton School, U. of PA,
Phila, PA 19104
http://www.jscottarmstrong.com
home phone 610 622 6480
Home address: 645 Harper Ave., Drexel Hill, PA 19026
Fax at school: 215 898 2534
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.sjdm.org/mail-archive/jdm-society/attachments/20070505/935e16bf/attachment-0001.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: unknown.gif
Type: image/gif
Size: 1241 bytes
Desc: not available
Url : http://www.sjdm.org/mail-archive/jdm-society/attachments/20070505/935e16bf/attachment-0001.gif
More information about the Jdm-society
mailing list