In the current issue of the NEJM, Serruys et al describe the results of a multicenter RCT comparing PCI with CABG for severe coronary artery disease: http://content.nejm.org/cgi/content/full/360/10/961. The trial, which was designed by the [profiteering] makers of drug-coated stents, was a non-inferiority trial intended to show the non-inferiority (NOT the equivalence) of PCI (new treatment) to CABG (standard treatment). Alas, the authors appear to misunderstand the design and reporting of non-inferiority trials, and mistakenly declare CABG as superior to PCI as a result of this study. This error will be the subject of a forthcoming letter to the editor of the NEJM.
The findings of the study can be summarized as follows: compared to PCI, CABG led to a 5.6% reduction in the combined endpoint of death from any cause, stroke, myocardial infarction, or repeat vascularization (P=0.002). The caveats regarding non-inferiority trials notwithstanding, there are other reasons to call into question the interpretation that CABG is superior to PCI, and I will enumerate some of these below.
1.) The study used a ONE-SIDED 95% confidence interval - shame, shame, shame. See: http://jama.ama-assn.org/cgi/content/abstract/295/10/1152 .
2.) Table 1 is conspicuous for the absence of cost data. The post-procedural hospital stay was 6 days longer for CABG than PCI, and the procedural time was twice as long - both highly statistically and clinically significant. I recognize that it would be somewhat specious to provide means for cost because it was a multinational study and there would likely be substantial dispersion of cost among countries, but it seems like neglecting the data altogether is a glaring omission of a very important variable if we are to rationally compare these two procedures.
3.) Numbers needed to treat are mentioned in the text for variables such as death and myocardial infarction that were not individually statistically significant. This is misleading. The significance of the composite endpoint does not allow one to infer that the individual components are significant (they were not) and I don't think it's conventional to report NNTs for non-significant outcomes.
4.) Table 2 lists significant deficencies and discrepancies between pharmocological medical management at discharge which are inadequately explained as mentioned by the editorialist.
5.) Table 2 also demonstrates a five-fold increase in amiodarone use and a three-fold increase in warfarin use at discharge among patients in the CABG group. I infer this to represent an increase in the rate of atrial fibrillation in the CABG patients, but because the rates are not reported, I am kept wondering.
6.) Neurocognitive functioning and the incidence of defecits (if measured), known complications of bypass, are not reported.
7.) It is mentioned in the discussion that after consent, more patients randomized to CABG compared to PCI withdrew consent, a tacit admission of the wariness of patients to submit to this more invasive procedure.
In all, what this trial does for me is to remind me to be wary of an overly-simplistic interpretation of complex data and a tendency toward dichotimous thinking - superior versus inferior, good versus bad, etc.
One interpretation of the data is that a 3.4 hour bypass surgery and 9 days in the hospital !MIGHT! save you from an extra 1.7 hour PCI and another 3 days in the hospital on top of your initial committment of 1.7 hours of PCI and 3 days in the hospital if you wind up requiring revascularization, the primary [only] driver of the composite endpoint. And in payment for this dubiously useful exchange, you must submit to a ~2% increase in the risk of stroke, have a cracked chest, risk surgical wound infection (rate of which is also not reported) pay an unknown (but probably large) increased financial cost, risk some probably large increased risk of atrial fibrillation and therefore be discharged on amiodarone and coumadin with their high rates of side effects and drug-drug interactions, while coincidentally risk being discharged on inadequate medical pharmacological management.
Looked at from this perspective, one sees that beauty is truly in the eye of the beholder.
This is discussion forum for physicians, researchers, and other healthcare professionals interested in the epistemology of medical knowledge, the limitations of the evidence, how clinical trials evidence is generated, disseminated, and incorporated into clinical practice, how the evidence should optimally be incorporated into practice, and what the value of the evidence is to science, individual patients, and society.
Subscribe to:
Post Comments (Atom)
I am reminded of this article in the NEJM in 2002:
ReplyDeletehttp://content.nejm.org/cgi/content/abstract/347/15/1143
HEPARIN PLUS ALTEPLASE COMPARED WITH HEPARIN ALONE IN PATIENTS
WITH SUBMASSIVE PULMONARY EMBOLISM.
In fellowship journal club, we ridiculed this article because the composite end-point was driven entirely by rescue use of alteplase. Thus, the entire article can be summarized as follows:
"Certain TPA now saves you from possible TPA later."
Good points. Number 3 especially bugs me because I had missed it the first time through. Looking back, I'm actually surprised that the editors at NEJM let them leave that in there.
ReplyDeleteAs for #1, Help me out here. I must have been out ice fishing the day this was discussed in medical school. My understanding is that generally speaking, equivalence trials should be analyzed with a 1-sided confidence interval and that noninferiority studies should be 2-sided, but I don't understand the specifics of why this is well enough to say why it was wrong in this case to go 1-sided.
With my (limited) background, it seems that this should have been designed as a superiority trial by the fine folks at Boston Scientific, comparing PCI and CABG in patients with low and/or intermediate SYNTAX scores.
To be statistically conservative, you get only .025 at each end of the confidence interval. If you are going to do a one-sided analysis, because you assume that any effect is going to be unidirectional, or because you don't care if it goes the other way, you can do a one-sided test, but you can't use the .025 from the other end of the interval as a free lunch. That is to say, if you're testing one-sided, you should use a .975 confidence interval - if you want to be conservative, that is, and favor the null.
ReplyDeleteThere is a GREAT figure, Figure 1 in the 2006 paper by Piaggio in JAMA - the CONSORT statement on the conduct and reporting of non-inferiority trials. I'll post a link to it in the article with point #1. If it weren't for this paper and this figure, I would have little understanding of non-inferiority trials.
Thanks for the comments!
Scott
Got it. Thanks. So basically the issue here is one-sided is fine, but they should have adjusted their that they should have adjusted their CI to 97.5%. Looking forward to reading that paper.
ReplyDeleteThanks for the information.
ReplyDelete