[Jdm-society] replications

[Robyn Dawes]

But David,
   
     That 5% tells you something precise.  And oook at your own last
paragraph.  You're extending beyond a particular study, whicvhis all that
the significance level or replication probability tells you about.  Wat
you're asking for is (to quope Meehl) "consistency across qualitatively
diverse contexts,  We have no STATISTICAL theory about that,. Only about
sampling in the same way from the same poipulaztion--in which case there is
no point doing it.
  Admittedly, people do screw up.  For example, the
"Shrunken" multiple correlation coefficient TELLS you what to expect in the
population as a whole.  (Little known math: the expected value of multiple
R-square with k predictive variables sampling from a population with a 0
value is k/( n - 1)  Subtract that from the sample R-squared, divide by 1
minus that value, and you get the Wherry-Lord formula.  (Yes, after a lot of
algebraic "muxing abound").
    But people often get a smaller value when they do an empirical split
half correlation.  Why?  Because the forgot they started our with k
predictors but after looking reduced the number to k', in effect giving
weights of 0 to k - k' of them.  And then the formula doesn't work, because
it's not the right formula.
   But there's no reason to replicate it empirically--unless you mistrust
the "apriori synthetic" (to use E. Kant's term) clout of mathematics.
    So you're all being 'post-modern," without, of course, realizing that.

Robyn      


On 5/7/07 11:15 AM, "David Budescu" <dbudescu at cyrus.psych.uiuc.edu> wrote:

> 
> Robyn
> 
> I think a more accurate statement is not that "they get it
> right" (which is rather vague and subjective), but rather
> that "they identify with satisfactory confidence those situations
> (including boundary cases) where the phenomenon is obtained)".
> 
> In my opinon  replications matter because not only because they
> can help dismiss chance results that are published (these false 5%
> rejections of the null), but also because they can help delineate the
> relevant parameter (broadly defined) space:
> 
> For example -- what are the boundaries of loss aversion, overweighting
> of low probability events, self enhancement judgments, underestimation
> of high probabilities. etc.
> 
> David
> 
> On Mon, 7 May 2007, Robyn Dawes wrote:
> 
>> Irwin,  What the "real" scientists do is to futch around until they get i
>> "right."  The multiple study requirement just adds "first and second and
>> third" studies, thereby wasting space and time.
>> 
>> Robyn
>> 
>> 
>> On 5/5/07 1:59 PM, "irwin-levin at uiowa.edu" <irwin-levin at uiowa.edu> wrote:
>> 
>>> Hi.
>>> I think this dialog is a great example of how we can use the SJDM list
>>> to address an issue that affects virtually all of us.
>>> A couple minor points to add:
>>>   1. A "failure to replicate" may sometimes be caused by going too far
>>> beyond the original operational definitions and boundary conditions.
>>> The Levin, Schneider & Gaeth '98 OBHDP paper, "All frames are not
>>> created equal" was all about that.
>>>   2. A number of major journals call for multiple experiments in one
>>> paper. I think that this can sometimes go too far by delaying
>>> publication of provocative results, but it does relate to the current
>>> issue in that the latter experiments in a series are often part
>>> replication and part extension of the earlier experiments.
>>> Irwin
>>> 
> 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> | David V. Budescu     |
> | Department of Psychology, University of Illinois |
> | 603 E. Daniel Street, Champaign, IL 61820  |
> | Tel (217) 333 6758 or (217) 840 1586   |
> | Fax (217) 244 5876 Email: dbudescu at uiuc.edu  |
> | http://www.psych.uiuc.edu/~dbudescu    |
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>