Needless to say, in spite of a dormant hope something so simple could prevent cancer, I was skeptical. Despite decades, perhaps eons of enthusiasm for the use of vitamins, minerals, and herbal remedies, there is, to my knowledge (please, dear reader, direct me to the data if this is an omission) no credible evidence of a durable health benefit from taking such supplements in the absence of deficiency. But supplements have a lure that can beguile even the geniuses among us (see: Linus Pauling). So before I read the abstract and methods to check for the level of statistical significance, the primary endpoint, the number of endpoints, and sources of bias, I asked myself: "What is the probability that taking a simple commercially available multivitamin can prevent cancer?" and "what kind of P-value or level of statistical significance would I require to believe the result?" Indeed, if you have not yet seen the study, you can ask yourself those same questions now.

I decided that I would require a P-value of less than 0.01 because I thought the probability of the alternative hypothesis (that vitamins reduce cancer) was low. This was admittedly arbitrary. But so is the standard P-value threshold of 0.05. The important thing is that the threshold is set a priori. That the authors chose a value higher than mine is of no consequence to me, so long as I set my own before knowing the actual results. I think this is a useful exercise, both because it forces one to think critically about the hypothesis ex ante and because it's fun! It's like a guessing game, and you get immediate feedback. And we know from decision making research that three things are required to calibrate expert decision making: 1.) Domain Specific Knowledge; 2.) Iterative practice; and 3.) prompt feedback.

So I was not surprised to find that the P-value for the [adjusted] Cox Hazard ratio was 0.04. (We have to read the text or go to Figure 2 to find that the Crude Log Rank P-value is = 0.05, i.e., NOT statistically significant.) One is left to wonder why you should need to adjust for baseline variables with 7000 persons in each group, but I will not belabor this point. Suffice it to say that this is a marginal result, and none of the lay press materials I perused mentioned this detail. Instead they focused on the the 8% reduction in cancer, musing at whether this is a large enough difference to be meaningful. Whether or not it is depends on whether it's an absolute or a relative difference. And since it's based on a HR, it's relative. So we need to see the actual data.I don't have the data necessary to reduplicate their Cox analysis, but Table 2 does give me the raw data on numbers of patients with cancer. (These do not account for duration, or time-to-event, so my analysis with these data is necessarily contrived and incomplete, but illuminating nonetheless.) Plugging the raw data into STATA, I find that the rate of cancer in the multivitamin (MVI) group is 17.6% and that in the placebo group is 18.8%. Whoa! What happened to 8%? That was a relative difference - the absolute difference is only 1.2% using the raw data. Suddenly, taking an MVI every day loses much of its luster.

But, there's more. To demonstrate how sensitive this P-value is to very small changes in outcomes, I ran the raw numbers and then added 5 and then 10 cancers to the placebo group. Below is the table that shows what happens to the P-value when these changes are made (the P-value of 0.06 is very close to the Crude Log Rank P-value of 0.05, even though I'm not using time-to-event data). The image below is clickable:

The statistical result is sensitive to just 5 more cancers in a group of 7000 patients, such that a marginal P-value could easily be pushed one way or another over the significance threshold. And, of course, cancer ascertainment is an imprecise endeavor. (So, we could look at mortality, which is more concrete, and which showed no difference.)

There is one more poignant set of results revealed in the article. But first, let me ask you: What do you think is the probability that the daily use of an MVI will influence the rates of epistaxis (nosebleeding) and hematuria (blood in the urine), and in what direction, increase or decrease? It turns out that MVI use in this study reduced hematuria (HR 0.91, P=0.02) and it increased epistaxis (HR 1.10, P=0.01). Fortunately, my a priori P-value for the overall results would have excluded these two findings as above the significance threshold, so I need not worry about them. But they are useful to demonstrate how, in a study of this size, spurious results are wont to arise. (See this future post about prespecified falsification endpoints.)

On this blog, I have described ad nauseum the importance of biological precedent and plausibility, multiple comparisons, etc, so I won't belabor these points again here. Suffice it to say that once these high-flying results are taken out of orbit for closer examination without spin, we see that multivitamins (and other supplements) create both expensive urine and expensive studies - and a lot of it just goes right down the drain.

Good analysis Scott. My own take is that Centrum Silver is composed of all artificial and synthetic ingredients. These do not penetrate the nucleus of cell. This brand was characterized in the book Comparative Guide to Dietary Supplements by Lyle McWilliams, as 4% absorbed.

ReplyDeleteAdditionally, in favor of a difference that could be shown with a potentially more effective multivitamin - the general public is NOT eating the NCI-required 5-9 servings of fruits and vegetables.

The RDA is known to be ridiculously low, all it can accomplish is to relieve our fear of scurvy and the like.

I founded a nonprofit annieappleseedproject.org providing information for people with cancer on natural strategies for risk reducation, recurrence, treatment and survivorship.

Thank you for putting into words what I have believed for years.

ReplyDeleteIf I didn't know better, I'd think Peter Bach was reading the blog:

ReplyDeletehttp://jama.jamanetwork.com/article.aspx?articleid=1391897

Also, a correction: the index article appeared in the November 14th print edition of JAMA. It was published online on October 17th.