- beware of studies with small sample sizes
- beware of studies with small effect sizes (delta)
- beware of multiple hypothesis testing and soft outcome measures
- beware of flexibility of designs (think Prowess/Xigris among others), definitions, outcomes (NETT trial), and analytic modes
Perhaps most importantly, he discusses the role that researcher bias may play in analyzing or aggregating data from research reports - the GIGO (garbage in, garbage out) principle. Conflicts of interest extend beyond the financial to tenure, grants, pride, and faith. Gone forever is the notion of the noble scientist in pursuit of the truth, replaced by the egoist climber of ivory and builder of Babel towers, so bent on promoting his or her (think Greet Van den Berghe) hypothesis that they lose sight of the basic purpose of scientific testing, and the virtues of scientific agnosticism.
Ioannodis also rails against entire (unnamed) fields of research saying that some of their results in aggregate are probably usable as a surrogate measure of the amount of bias in the field. So, assume that alternative and complementary medicine, and herbals and supplements, and nutritional interventions and all that jazz that has been all the rage for a decade now - suppose that it is all poppycock and the real truth is that none of it works - if you look in aggregate at the data in that field, any net effect is likely to reflect nothing more than the bias in that field. What a refreshing concept. Cheers.
For my own part, I still taste saltiness and bitter almonds whenever I think of Xigris, that fallen angel of sepsis which was dispatched to hell last month after a [final] study showed no efficacy. I was duped for a full decade on that one, perhaps because I gave too little weight to a change in the formulation of the drug mid-trial. Shame on me. Live and learn.
In reading his article, I am reminded also of outright fraud in science and several salient and contemporaneous examples. Hwang Woo Suk of South Korea basically faked a whole program of stem cell research, results of some of which studies were published in that pinnacle of peer-reviewed research, "Science"; Mark Hauser of Harvard continues to face investigation because irregularities in his research (some also published in Science); and most recently, the author of a This 2011 Study of Prejudice in Science has been accused of widespread fraud and scientific "misconduct on an astonishing scale."
So much for the peer-review process and impact factors. They're no match for ego-maniacal cheaters hell bent on tenure, grants, and manuscripts no matter the costs.....or consequences.