logoalt Hacker News

derbOactoday at 1:35 PM1 replyview on HN

Experimental designs are critical for obvious reasons but they have a few critical flaws, that mostly all reduce to the fact you can't randomize manipulations with everything. Whether it be due to ethics or practical constraints, you can't conduct a RCT all the time.

This can be more subtly critical than it might seem, in that even if you can manipulate some proxy, often that proxy is insufficient in actually representing the phenomenon of interest, or the conditions under which they actually occur.

I often use the example of videogames and aggression. There were plenty of experimental studies of this but it was always questionable whether lab-induced anger is the same thing as, say, the sort of violence we generally are concerned about societally.

I generally have tried to teach students that experimental designs when done right provide powerful causal evidence of something, but often with limited generalizability; observational designs in contrast provide powerful generalizable evidence of some kind of association, but often with limited certainty about the causal pathways involved.

I've been in a department that was rabidly experimental in its focus and it always seemed sort of short-sighted, because people were idolizing RCTs with proxy manipulations that had questionable generalizability to the real-world phenomena they were trying to model.

Ideally you'd bring both experimental and observational evidence to bear on a question. Your conclusions should be robust to different types of designs.


Replies

CGMthrowawaytoday at 4:40 PM

Yes. Even conceding every point made in this article, RCTs are still the gold standard. The article points out that RCTs are often impractical, expensive or ethically challenging. Well, so is the gold standard of anything.

OP suggests that alternative methods like target trial emulation, propensity scoring and double machine learning can be used to approximate the conditions of an RCT using existing data. In saying so he gives away the tell, which is that RCTs are the standard being aspired to.

Observational data may well be undervalued. But RCTs are still the gold standard.