We have a dual problem in medical research: a.) of conceiving alternative hypotheses which cannot be confirmed in large trials free of bias; and b.) not being able to replicate the findings of positive trials. What are the reasons for this?
Trials can be negative either because of 1.) a lack of statistical power, or; 2.) because they are testing therapies that just don't work, often on the basis of poorly conceived or naive hypotheses. Studies can be impossible to replicate because: 3.) the results represent a "false positive" or type 1 error, or; 4.) they had inherent bias which drove the positive results. Since Warren Buffet got me thinking about this post, I will extend an analogy:
- underpowering (type II errors) is analogous to a start-up that can't get enough venture capital
- hypotheses that just don't work are like the dot com bubble company pets.com
- false positive results (type I errors) are like pets.com before the bubble bursts
- inherent study bias is analogous to a company that is cooking the books like Enron (think intensive insulEnron therapy).