Written by: Brent Donnellan
Primary Source: The Trait-State Continuum
I loved Simine’s blog post on flukiness and I don’t want to hijack the comments section of her blog with my own diatribe. So here it goes…
I want to comment on the suggestion that researchers should propose an alternative theory to conduct a useful or meaningful close/exact/direct replication. In practice, I think most replicators draw on the same theory that original authors used for the original study. Moreover, I worry that people making this argument (or even more extreme variants) sometimes get pretty darn close to equating a theory with a sort of religion. As in, you have to truly believe (deep in your heart) the theory or else the attempt is not valid. The point of a direct replication is to make sure the results of a particular method are robust and obtainable by independent researchers.
Original authors used Theory P to derive Prediction Q (If P then Q). This is the deep structure of the Introduction of their paper. They then report evidence consistent with Q using a particular Method (M) in the Results section.
A replicator might find the theoretical reasoning more or less plausible but mostly just think it is a good idea to evaluate whether repeating M yields the same result (especially if the original study was underpowered).* The point of the replication is to redo M (and ideally improve on it using a larger N to generate more precise parameter estimates) to test Prediction Q. Some people think this is a waste of time. I do not.
I don’t see how what is inside the heads of the replicators in terms of their stance about Theory P or some other Theory X as relevant to this activity. However, I am totally into scenarios that approximate the notion of a critical test whereby we have two (or more) theories that make competing predictions about what should be observed. I wish there were more cases like that to talk about.
* Yes, I know about the hair splitting diatribes people go through to argue that you literally cannot duplicate the exact same M to test the same prediction Q in a replication study (i.e., the replication is literally impossible argument). I find that argument simply unsatisfying. I worry that this kind of argument slides into some postmodernist view of the world in which there is no point in doing empirical research (as I understand it).
Latest posts by Brent Donnellan (see all)
- My View on the Connection between Theory and Direct Replication - April 9, 2015
- How Do You Feel When Something Fails To Replicate? - March 13, 2015
- (Hopefully) The Last Thing We Write About Warm Water and Loneliness - November 19, 2014