If you didn't care to know anything about finance, and you hired a financial adviser (paid hourly, not through commissions, of course) you would be happy to have him simply tell you to invest all of your assets into a Vanguard life cycle fund. But you may then be surprised that a different adviser told one of your contemporaries that the approach was oversimple and that you should have several classes of assets in your portfolio that are not included in the life cycle funds, such as gold or commodities. In light of the discrepancies, you may conclude that to make the best economic choices for yourself, you need to understand finance and the data upon which the advisers are basing their recommendations.

Making medical decisions optimally is akin to making economic decisions and is founded on a simple framework: EUT, or Expected Utility Theory. To determine whether to pursue a course of action versus another one, we add up the benefits of a course multiplied by their probability of accruing (that product is the positive utility of the course of action) and then subtract the product of the costs of the course of action and their probability of accruing (the negative utility). If utility is positive, we pursue a course of action, and if options are available, we pursue the course with the highest positive utility. Ideally, anybody helping you navigate such a decision framework would tell you the numbers so you could do the calculus. Using the finance analogy again, if the adviser told you "Stocks have positive returns. So do bonds. Stocks are riskier than bonds" - without any quantification, you may conclude that a portfolio full of bonds is the best course of action - and usually it is not.

I regret to report that that is exactly what clinical practice guideline writers do: provide summary information without any numerical data to support it, leaving the practitioner with two choices:

Making medical decisions optimally is akin to making economic decisions and is founded on a simple framework: EUT, or Expected Utility Theory. To determine whether to pursue a course of action versus another one, we add up the benefits of a course multiplied by their probability of accruing (that product is the positive utility of the course of action) and then subtract the product of the costs of the course of action and their probability of accruing (the negative utility). If utility is positive, we pursue a course of action, and if options are available, we pursue the course with the highest positive utility. Ideally, anybody helping you navigate such a decision framework would tell you the numbers so you could do the calculus. Using the finance analogy again, if the adviser told you "Stocks have positive returns. So do bonds. Stocks are riskier than bonds" - without any quantification, you may conclude that a portfolio full of bonds is the best course of action - and usually it is not.

I regret to report that that is exactly what clinical practice guideline writers do: provide summary information without any numerical data to support it, leaving the practitioner with two choices:

In this recent and excellent Summary Guideline for Mechanical Ventilation in Adults with ARDS, the authors do not, for any recommendation of any strength, tell the reader the absolute benefit of the recommended interventions or the uncertainty around the estimate of the benefit (in terms of a confidence interval or other concrete measure). This dichotomization of interventions into "works" and "doesn't work" or "recommended" and "not recommended" does not allow the clinician to use a sound decision making framework for evaluating a course of action and forces him back to the data themselves, rendering the guideline a middleman of dubious utility.

What we want to know, when deciding whether to use prone positioning for a patient with moderate to severe ARDS is what the absolute benefit is, the degree of uncertainty of the estimate, and the costs and other negative utilities of the maneuver. Unfortunately, we are not told these numerical values and must search for them ourselves. A meta-analysis of 8 trials showed an absolute reduction in mortality with this therapy of 6% (RR 0.9), with a meta-analytic 95% confidence interval of 0.82-0.98 on a relative risk scale, which corresponds to about 1%-9% on an absolute scale. Ignoring the problems with the prone positioning saga I have previously chronicled, we can take these numbers at face value and conclude in binary fashion as the guideline writers have, that prone positioning is something we ought to be doing for patients with ARDS since the costs and complications appear to be much lower than a 6% risk of death. Well and good.

But suppose the patient for whom we are considering prone positioning weighs 170 kilograms and had a difficult airway when originally intubated. Suppose also that a perspicacious clinician has reasoned that he may not have ARDS at all, but rather atelectasis from obesity ("fat-alectasis") that, along with tissue density, is being mistaken for "bilateral opacities" on his CXR. This clinician thinks we need to probability adjust for the uncertainty of the diagnosis of ARDS, and she also worries that the risks are higher in this individual who could have a catastrophe if an accident happens during his turning. She further wonders if this patient would have even been included in the trials (done ins Europe) which were included in the meta-analysis, and if her uncertainty of the 6% estimate of efficacy should be increased if he did not meet inclusion criteria for some or all of the trials. How should all this uncertainty be resolved? What should she do?

One heuristic she may employ is one I describe on Status Iatrogenicus that I call the Number Needed Not to Treat to Harm (NNNTTTH) heuristic. Numerically identical to the number needed to treat (NNT) it is the number of patients you can

*fail*to treat before expecting to accrue a harm from omission of the therapy. In the simplest case, ignoring the aforementioned caveats about enhanced uncertainty in the case of this patient, the clinician could take the summary estimate of 6% from the meta-analysis and calculate a NNNTTTH=1/0.06 ~17. She can, in general, fail to use prone positioning 17 times in a patient in whom it is generally indicated before she fails to save a life with the therapy. Note also that if she probability adjusts the diagnosis in this case by 50% because of uncertainty about the diagnosis (50% of such patients don't actually have ARDS and cannot benefit from prone positioning), the NNNTTTH doubles to 34 patients like this one. With this number in mind, she may think "I bet that one in 34 times (or more) when we prone a patient such as this, the ETT will be dislodged and we will have a catastrophe on our hands trying to reintubate him." (She may also attempt to do a back-of-the-envelope estimate of the probability of death and injury during such an episode With the use of explicit numbers, she can work through some estimation of whether to use the therapy in a particular case.
Guideline writers, rather than using vague, abstract terminology, however standardized (e.g., GRADE), would do well to report estimates of effect size and measures of uncertainty around those effect sizes. Otherwise, just cut the middleman and make a bibliographic list of references we can use to find the source data ourselves.

Better articulated than I could: http://www.annualreviews.org/doi/full/10.1146/annurev-statistics-010814-020148

ReplyDeleteRisk (probability) and uncertainty (confidence intervals)