[Jdm-society] Replications

[acr.martins]

I must say that I completely agree with Mike Doherty, at least about dissertations in experimental areas. Something similar might be difficult to implement in purely theoretical subjects (I am thinking of things like theoretical Physics or, even worse, Mathematics), where simply making the same calculations presented in an old paper is a task too easy, once the original author has paved the way. That is, unless the student finds a new way of proving the same thing. And that would be, quite often, asking too much. For everyone else, the idea that every Master's dissertation should be a replication seems perfect and it would make the results in every area far more reliable.

And it is really necessary! On the reliability of the reported results as a whole, when one conducts no experience, I have made a few simple but disturbing calculations myself, by assuming a Bayesian reader who tries to decide between two theories. The reader does not perform experiments, only read descriptions of experiments done by others. If the reader includes in the analysis the possibility of error and deception, under some circumstances, no certainty is ever reached, even in the limit of infinite articles read.

Victor Palmer has extended that result for the case where the reader only uses replicated results as his basis. That seemed to solve the problem, but his analysis is only valid under some special circumstances. Still, it makes it very clear the need for replication.

If anyone is interested in those results, the references are

Martins, André C. R. (2005). Deception and Convergence of Opinions. JASSS-The Journal of Artificial Societies and Social Simulation, v. 8, n. 2.

Palmer, Victor (2006) Deception and Convergence of Opinions Part 2: the Effects of Reproducibility. JASSS-The Journal of Artificial Societies and Social Simulation, v. 9, n. 1.


André C. R. Martins
Escola de Artes, Ciências e Humanidades
Universidade de São Paulo
Brazil


> My first week as a faculty member at Bowling Green was in September of 1965. The Ohio 
> Board of Regents had approved our PhD program in August. That first week was a week-long 
> faculty meeting, dedicated almost exclusively to putting the meat onto the bones of the 
> approved doctoral proposal. For a few years after that, I kept promoting the idea that an 
> acceptable Master's thesis should be a literal replication, to the extent permitted by the 
> quality of the Method section of the paper that had reported the experiment being replicated. 
> To the best of my recollection, not a single person agreed! I still think it is a good idea, and it 
> would not create another requirement (I comletely concur with Jon's terror of adding yet 
> another requirement) and it would likely get the students into an active research mode much 
> sooner.
> 
> There is a problem. A couple of years later, I taught a seminar in cognitive, and had as a 
> requirement that the students select an important study that could be replicated with the 
> resources at hand, but they only had to run a single subject. The purposes of doing so were 
> to get them to read more closely than they ever had before, and to get them to see the 
> imprtance of writing clear and complete method sections. Every student concluded that they 
> could not do a truly literal replication without contacting the author(s) of the paper!
> 
> Perhaps stressing the need for replication might have a side effect of more complete 
> writeups?  
> 
> Mike Doherty