Tuesday, August 11, 2009

Vertebroplasty: Absence of Evidence Yields to Evidence of Absence. It Takes a Sham to Discover a Sham but how will I Get a Sham if I Need One?

"When in doubt, cut it out" is one simplified heuristic (rule of thumb) of surgery. Extension (via inductive thinking) of the observation that removing a necrotic gallbladder or correcting some other anatomic aberration causes improvement in patient outcomes to other situations has misled us before. It is simply not always that simple. While it makes sense that arthroscopic removal of scar tissue in an osteoarthritic knee will improve patients' symptoms, alas, some investigators had the courage to challenge that assumption, and reported in 2002 that when compared to sham surgery, knee arthroscopy did not benefit patients. (See http://content.nejm.org/cgi/content/abstract/347/2/81.)

In a beautiful extension of that line of critical thinking, two groups of investigators in last week's NEJM challenged the widely and ardently held assumption that vertebroplasty improves patient pain and symptom scores. (See http://content.nejm.org/cgi/content/abstract/361/6/557 ; and http://content.nejm.org/cgi/content/abstract/361/6/569 .) These two similar studies compared vertebroplasty to a sham procedure (control group) in order to control for the powerful placebo effect that accounts for part of the benefit of many medical and surgical interventions, and which is almost assuredly responsible for the reported and observed benefits of such "alternative and complementary medicines" as accupuncture.

There is no difference. In these adequately powered trials (80% power to detect a 2.5 and a 1.5 point difference on the pain scales respectively), the 95% confidence intervals for delta (the difference between the groups in pain scores) were -0.7 to +1.8 at 3 months in the first study and -0.3 to + 1.7 at 1 month in the second study. Given that the minimal clinically important difference in the pain score is considered to be 1.5 points, these two studies all but rule out a clinically significant difference between the procedure and sham. They also show that there is no statistically significant difference between the two, but the former is more important to us as clinicians given that the study is negative. And this is exactly how we should approach a negative study: by asking "does the 95% confidence interval for the observed delta include a clinically important difference?" If it does not, we can be reasonably assured that the study was adequately powered to answer the question that we as practitioners are most interested in. If it does include such a value, we must assume that for us given our judgment of clinical value, the study is not helpful and essentially underpowered. Note also that by looking at delta this way, we can determine the statistical precision (power) of the study - powerful studies will result in narrow(er) confidence intervals, and underpowered studies will result in wide(r) ones.

These results reinforce the importance of the placebo effect in medical care, and the limitations of inductive thinking in determining the efficacy of a therapy. We must be careful - things that "make sense" do not always work.

But there is a twist of irony in this saga, and something a bit concerning about this whole approach to determining the truth using studies such as these with impeccable internal validity: they lead beguillingly to the message that because the therapy is not beneficial compared to sham that it is of no use. But, very unfortunately and very importantly, that is not a clinically relevant question because we will not now adopt sham procedures as an alternative to vertebroplasty! These data will either be ignored by the true-believers of vertebroplasty, or touted by devotees of evidence based medicine as confimation that "vertebroplasty doesn't work". If we fall in the latter camp, we will give patients medical therapy that, I wager, will not have as strong a placebo effect as surgery. And thus, an immaculately conceived study such as this becomes its own bugaboo, because in achieving unassailable internal validity, it estranges its relevance to clinical practice insomuch as the placebo effect is powerful and useful and desireable. What a shame, and what a quandry from which there is no obvious escape.

If I were a patient with such a fracture (and ironically I have indeed suffered 2 vertebral fractures [oh, the pain!]), I would try to talk my surgeon into performing a sham procedure (to avoid the costs and potential side effects of the cement).....but then I would know, and would the "placebo" really work?

5 comments:

  1. I also enjoyed reading that article. You just don't see a lot of sham procedure studies these days. There's no way if I tell somebody I'm giving them NSAIDs, a back brace, and physical therapy that they will think that is anywhere near as effective as a space age polymer injected into their vertebral body. Most of my patients just don't care about the NEJM. Will our new national health care system continue to pay for this procedure that has no proven efficacy?

    ReplyDelete
  2. I have taken care of 2 patients with infected vertebroplasties. 1 patient had an underlying osteomyelitis, which looked like a collapsed vertebrae. This was discovered AFTER the cement was injected. The other pt probably was infected at the time of injection. What a mess!! There is no way to reverse this procedure, and so they are now both on lifelong antibiotic suppression. Another bad outcome is the injection of cement into the disk space. This causes a huge inflammatory rxn which looks like discitis. not surprisingly all these poor outcomes lead to MORE pain.

    ReplyDelete
  3. I get what you're saying - and agree for the most part. However, this also impinges on the issue of blinding. If you don't do something that seems like the active intervention, other care might change as well (e.g. use of NSAIDs, PT, etc). You are (perhaps unwittingly) arguing for more of the hot approach to research: "clinical effectiveness research." So, how woudl YOU design the study? Instead of 2 studies, do one study with 3 arms - usual care, sham, and vetrebroplasty? Difference between sham and usual care (if any) is the placebo effect and this is subtracted from apparent effect of active intervention over usual care? I belive you have railed against the inclusion of a UC arm in other trials - say tidal volume in ALI/ARDS?

    ReplyDelete
  4. Thank you all for your comments.

    My former contrarian colleague Dr. O'Brien has made some good points as usual.

    There is the battle between efficacy and effectiveness research, and these studies were clearly designed to test the former. That is, they ruled out a biological mechanism other than placebo for any apparent benefit of vertebroplasty. And this is a very useful and commendable effort. But I still think it leaves us in a quandry.

    I agree with Jim about the need for blinding to maximize internal validity, and I do question whether it would be useful to have a third arm where "usual non-invasive care" is compared to vertebroplasty and sham, because of these potential differences in care that might result because this group is not blinded. But again, if the usual non-invasive care group fared better, regardless of differences in care, I still think that is a very useful result, and so would be the converse result.

    The inclusion of a third group does pose several logistical problems and I don't know if I generally favor it.

    This is a very interesting topic and I don't know the answers. But I like thinking about the questions.

    Thanks again for your comments!

    ReplyDelete
  5. This research on placebo, as serious as it is, reminds me to give a new twist to an old joke.

    In the Marx Brothers “Duck Soup,” Rufus T. Firefly, played by Groucho, is handed the Freedonia cabinet’s treasury report and states, “Why, a child of 4 could understand this report. Run out and find me a 4-year-old child — I can’t make head or tail of it.”

    Not that I'm suggesting we run out and solicit 4-year-olds to design double blind RCTs.

    Just that thinking about small children can shed new light on conundrums like placebo and healing.

    Like placebo, early childhood is both familiar and mysterious.

    Adults have experienced both, but we simply can’t remember what it was like to be our own magic healers at an age younger than 5 or 6.

    Working in an area of clinical remediation (and symptom relief) for something as subtle and intangible as organic/vegitative balance-related issues has left me believing that we adopt an incremental progression from "know-cebo" to placebo. Yet we have precious little understanding of this gradual morphosis. I can say with assurity that when I have my clients adopt an o-2-b-4 again playful mentality they are in a more susceptible position to heal. For instance, when children are playing, they know they are just playing. Yet play is a very serious business that affords us the luxury of exploring the world with unfettered imagination.

    Playful immersion in freely conjured hypothetical worlds is what teaches us how to make sense of the real one.

    Indeed it gives us the capacity to appreciate more fully “counterfactual” situations such as "How Do I Get a Sham if I Need One" and then calculate the probabilities of alternative courses of action.

    ReplyDelete

Note: Only a member of this blog may post a comment.