[Jdm-society] Replications, etc

[Terry Connolly]

Hi all:
One thought on the replications issue: we might want to try making 
more defensible claims in the first place. That is, it might help if 
we reported our results as demonstrations not of universal rules but 
of existence theorems -- not "The study shows that Treatment A 
affects Variable B in the following way" ( a general rule) but 
"Treatment A affected Variable B in the following way" (reminding us 
that we are reporting a specific event, with specific populations, 
treatments, incentives, etc, -- the host of empirical peculiarities 
that make it interesting question whether or not we will get a 
similar event when any one of them is different). If you think that's 
what we are doing now, try examining the change in verb tenses 
between most Method sections and their subsequent Discussions. The 
Methods sections are all careful past tense ("These people did this 
in these circumstances"), the Discussion section generally slips into 
totally unjustified continuous present ("People do this in these 
circumstances"). It's these latter claims that "fail to replicate". 
That's maybe because we shouldn't have made them in the first place.

If we confine ourselves to careful, past-tense specifics, we get 
{Instance 1: A happened}; {Instance 2: B happened}; {Instance 3: A 
again}; and so on.  Given a reasonable pool of such specifics, we can 
then start to infer the domains of As, Bs, and the like, and state 
some (tentative) generalizations and their boundary conditions. But, 
given the competence of each set of observations, note that we don't 
get "successful" and "unsuccessful" replications, we get, well, 
specific instances, each valuable whether it's an A or a B or 
something else altogether.

For instance: The first study (probably by K&T) says: Here's an 
instance (described in detail) in which some people showed 
overconfidence, loss aversion, sunk cost honoring, non-Bayesian 
revision, decoy effect or whatever. No-one, I assume, thinks we can 
now assert that people are everywhere and always overconfident, 
loss-averse, sunk cost trapped, etc. What we have is an existence 
theorem: There exist circumstances under which some people are 
overconfident, loss averse... Since we don't (yet) believe this is a 
general rule, let's document those circumstances as carefully as we 
can.  The next study (perhaps a modest extension, say, to a different 
campus) finds either the same effect or the reverse or neither. Again 
assuming competence, we've learned something: The effect is or is not 
robust to that specific change -- it works for Stanford-smart 
students, but not for Podunk State students. Now to further specific 
changes: more expert subjects, larger incentives, different 
instantiations, new measures, longer time periods, as we grope around 
the possible boundary conditions for over-, under- and accurate 
confidence judgments. In this language, a "replication" is either (a) 
a demonstration of competence, or (b) a demonstration that the effect 
boundaries are at least broad enough to encompass the inevitable 
small changes between Study 1 and Study 2. Disagreement between the 
two studies is evidence that the two span the effect's boundaries. If 
studies 2 through N all fall outside the boundary, then the suspicion 
grows that Study 1 was either flawed or represents an effect of very 
small domain (or, of course, sampling error).

Terry

Terry Connolly
FINOVA Professor of Management and Policy
The Eller College
University of Arizona
Tucson, AZ 85721
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.sjdm.org/mail-archive/jdm-society/attachments/20070507/63eadd32/attachment.htm