Tuesday, April 4, 2017

Tipping the Scales of Noninferiority: Abbott's "Emboshield and Xact Carotid Stent System"

I just stumbled across this and think it's worth musing over it a bit.  The recently published ACT I trial by Rosenfield et al is a noninferiority trial of an already approved device, the "emboshield embolic protection system" used in conjunction with the "Xact carotid stent system" both proprietary devices from Abbott.  I'm scrutinizing this trial (and others) to determine if adequate justification is given for the noninferiority hypothesis around which the trial is designed.  One thing I'm looking for is  evidence that there are clear secondary advantages of the novel or experimental therapy that justify accepting some degree of worse efficacy, compared to the active control, which falls within the prespecified margin of noninferiority.  This is what the authors (or their ghosts) write in the introduction:
"Most carotid revascularization procedures in the United States are carotid endarterectomies performed for the treatment of asymptomatic atherosclerotic disease. Revascularization is also performed by means of stenting with devices to capture and remove emboli (“embolic protection” devices).3,4 In the Carotid Revascularization Endarterectomy versus Stenting Trial (CREST), no significant difference was found between carotid endarterectomy and stenting with embolic protection for the treatment of atherosclerotic carotid bifurcation stenosis with regard to the composite end point of stroke, death, or myocardial infarction.5 CREST included both symptomatic and asymptomatic patients, and it was not sufficiently powered to discern whether the carotid endarterectomy and stenting with embolic protection were equivalent according to symptomatic status. The primary aim of the Asymptomatic Carotid Trial (ACT) I was to compare the outcomes of carotid endarterectomy versus stenting with embolic protection in patients with asymptomatic severe carotid-artery stenosis who were at standard risk for surgical complications."
That's a mouthful, to say the least, and probably ought to be expectorated.


The CREST trial was published in 2010, and it was a superiority trial designed to show (click for clinicaltrials.gov link) that the stent system used was superior to carotid endarterectomy (CEA) for the primary outcome of death, stroke, etc.  A little history is in order, since CREST was registered on www.clinicaltrials.gov in 2000.  It was in 1998 that two large trials showed the apparent efficacy of CEA for the prevention of stroke in patients with symptomatic carotid narrowing greater than 70%.  It was about the same time epoch that cardiac stents were gaining widespread use for the treatment of CAD, and being compared to the gold standard of CABG.  I am going to infer without investigating further that carotid stenting came on the heels of the success of coronary stenting, even though mechanistically they might not be comparable.

CREST enrolled 2500 patients to fulfill 90% power requirements to demonstrate a 1.2% absolute rate of difference in the primary outcome, corresponding to a hazard ratio of .54 in favor of stenting or 1.49 in favor of CEA.  (Interestingly, they described the alternative hypothesis as "the two treatments differ".)  The results of CREST were somewhat disappointing for Abbott, which presumably used a superiority design on purpose:  in the 4-year period of follow-up, the absolute difference in the primary outcome was 0.4% in favor of CEA with a 95% CI of -1.7% to +2.6%, a result that is not inconsistent with the null at the prespecified alpha (or almost any alpha!).  The disappointment comes about because, even though the result is firmly centered on zero with a relatively narrow and informative confidence interval, all you can claim is that you did not find a difference between the stent and CEA - you cannot claim that your stent is equivalent or noninferior to CEA because that would have required a noninferiority hypothesis from the outset, at least formally.  On the other hand, I can evaluate that confidence interval and say, "well if there's any difference between the two, it's likely to be smaller than 2.6%."

OK, now back to their justification for the ACT I trial:
"CREST included both symptomatic and asymptomatic patients, and it was not sufficiently powered to discern whether the carotid endarterectomy and stenting with embolic protection were equivalent according to symptomatic status."
Yes, the CREST trial was powered for the overall population enrolled, not for subgroup analyses or interactions, nor was it powered for a different design, namely an equivalence design.  But subgroup analyses in CREST did not even show a trend for an interaction between treatment effect and symptom status (Table 3):


They go on:
"The primary aim of the Asymptomatic Carotid Trial (ACT) I was to compare the outcomes of carotid endarterectomy versus stenting with embolic protection in patients with asymptomatic severe carotid-artery stenosis who were at standard risk for surgical complications."
They want to limit the ACT I trial to just asymptomatic patients because - well, why?  Because there are more asymptomatic than symptomatic patients, i.e., a greater market share, and because those patients may be far less motivated to get surgery since the NNT is smaller and the NNH larger with CEA for asymptomatic patients.  The implication for the new trial however, is that asymptomatic patients will have a lower event rate, meaning that if you study only them, you will need a much larger sample size than CREST where you had symptomatic and asymptomatic patients.

But there's more to that choice.  Noninferiority trials differ from superiority trials because in the latter, if the design specifies an inadequate does of the therapy being tested, the bias is towards the null, or finding no difference - a potential type II error.  In a noninferiority trial, inadequate dosing of the active control biases results towards the alternative hypothesis - that the new therapy is noninferior to the proven active control - a type I error.  So, in a noninferiority trial, investigators must assure that the active control is given at a maximal dose.  A corollary of this is related to the constancy assumption which will be discussed below.  Say you have a potent therapy administered at maximal dose (e.g., CEA) but you're administering it to a patient population that cannot benefit or can benefit very little - say patients with asymptomatic carotid stenosis.  Ah, you see!  Studying asymptomatic patients diminishes any efficacy of active control making it more likely that a placebo will not be significantly different than it, and this effect will be more pronounced with a wider margin of noninferiority.  This is yet another way that limiting the noninferiority trial to asymptomatic patients increases the rate of a type I error, favoring the false declaration of noninferiority of the "Emboshield".  Indeed, I'm not aware if this particular theoretical twist has been previously described in the literature.  Thus, we shall give it a name: "regression to the null".

So, how was the sample size for ACT I determined?
"If the upper limit of the one-sided 95% confidence interval of the difference in event rates between stenting and endarterectomy was 3 percentage points or lower (prespecified noninferiority margin), stenting was considered to be noninferior to endarterectomy.6 The study was originally designed to enroll 1658 patients, with 80% power to detect the noninferiority margin at P=0.05."
We begin complaining about lack of power of CREST, we endeavor to enroll patients with an expected lower baseline event rate, then we cut the sample size by 35%!  But of course, this is a wily thing to do, because they have that narrow confidence interval from CREST with an upper bound of:    2.6%, conveniently close to the upper bound selected for the noninferiority margin (which is not at all justified) in ACT I.  And note that, when we were looking for superiority, we sought to find a 1.2% absolute advantage for Emboshield, and we certainly would have taken any result on the order of 1.2% or 0.8% that was statistically significant, and say that the shield was superior.  But when we design the noninferiority trial, we are now willing to say that up to 3% is clinically negligible or within our "zone of indifference".

There are several interesting things here.  Since ACT I showed noninferiority with the 3% margin, we have added nothing to what we already know from inspecting the confidence interval from CREST, although if we combine the two trials, we have even greater statistical precision.  The main benefit of this trial is now we can formally say that Emboshield and Xact are noninferior to CEA in asymptomatic patients.

Noninferiority trials have several inherent assumptions and prerequisites, one principal among them  called the "constancy assumption."  One tenet of the constancy assumption is that noninferiority trials that use active controls that were proved superior in prior trials must assume that the patient characteristics and trial milieu are constant from the time the active control was proved superior to placebo to the time the noninferiority trial is performed.  The superiority trials from CEA were conducted 20+ years ago.  Much has changed in that time, especially in terms of adjuvant medical therapies such as treatments for cholesterol, diabetes, hypertension, and antiplatelet agents.  (Likewise, one could say that surgical treatment techniques have improved in parallel.)  It is quite possible that the constancy assumption is violated in noninferiority trials of CEA, and if it is, we are inviting bio-creep, or a degradation of the presumed superiority to placebo of treatments considered noninferior in successive generations of noninferiority trials.  If CEA is no longer superior to medical management, the ACT I trial has shown only that the "Emboshield" (Lord, that name begs derision) is noninferior to placebo, since all patients now presumably receive optimized medical management with stenting and surgery both.

Finally, note that this trial does not fully satisfy another basic tenet of noninferiority trials: namely that there be secondary benefits to the experimental therapy.  If we are willing to say that some patients would just to like to avoid surgery, I suppose I can't disagree with that, but note that this trial enrolled only patients with "standard risk for surgical complications" and that one of the major risks of both CEA and stenting are peri-procedural stroke (see Table 3 from CREST above - everything is numerically higher with the stent).  Alas, the major secondary advantage with the Emboshield is that Abbott profits from it, as do vascular IR types who wish to draw market share from the surgeons.

To summarize and be judicious, the data do suggest that any difference between Emboshield and CEA in populations like the one studied are likely to be small (if small is less than 3% absolute), assuming there are not other biases at work here such as resulting from lack of blinding/ascertainment, placebo effects (which are greater with surgery), fraud and other nonsense that I have failed to detect.  I'm left both amused and bemused.  I'm amused because of how sly these companies are in terms of manipulating the accepted framework of trial design to achieve their ends, and how often they and those that review the manuscripts don't understand the nuance or don't care about it.  And I'm bemused because in the back of my mind, I wonder if CEA would be shown to be efficacious in today's milieu of advanced adjuvant medical therapies for atherosclerotic vascular disease, especially with PCSK9 inhibitors on the horizon.


No comments:

Post a Comment

Note: Only a member of this blog may post a comment.