In the Dark

 I haven’t put anything in the Bad Statistics  file for a while, so I thought I’d put this interesting little example up for your perusal.

Although my own field of modern cosmology requires a great deal of complicated statistical reasoning, cosmologists have it relatively easy because there is not much chance that any errors we make will actually end up harming anyone. Speculations about the Anthropic Principle or Theories of Everything are sometimes  reported in the mass media but, if they are, and are garbled, the resulting confusion is unlikely to be fatal. The same can not be said of the field of medical statistics. I can think of scores of examples where poor statistical reasoning has been responsible for shambles in the domain of public health.

Here’s an example of how a relatively simple statistical test can lead to total confusion. In this version, it is known as Simpson’s Paradox.

 A standard thing to do in a medical trial is to take a set of patients suffering from some condition and divide them into two groups. One group is given a treatment (T) and the other group is given a placebo; this latter group is called the control and I will denote it T* (no treatment).

To make things specific suppose we have 100 patients, of whom 50 are actively treated and 50 form the control.  Suppose that at the end of the trial for the treatment, patients can be classified as recovered (“R”) or not recovered (“R*”).  Consider the following outcome, displayed in a contingency table:

 Clearly the recovery rate for those actively treated (40%) exceeds that for the control group, so the treatment seems at first sight to produce some benefit.

 Now let us divide the group into older and younger patients: the young group Y contains those under 50 years old (carefully defined so that I would belong to it) and Y* is those over 50.

 The following results are obtained for the young patients.

The older group returns the following data: 

 For each of the two groups separately, the recovery rate for the control exceeds that of the treated patients. The placebo works better than the treatment for the young and the old separately, but for the population as a whole the treatment seems to work better than the placebo!

This seems very confusing, and just think how many medical reports in newspapers contain results of this type: drinking red wine is good for you, eating meat is bad for you, and so on. What has gone wrong?

 The key to this paradox is to note that many more of the younger patients are actually in the treatment group than in the non-treatment group, while the situation is reversed for the older patients. The result is to confuse the effect of the treatment with a perfectly possible dependence of recovery on the age of the recipient. In essence this is a badly designed trial, but there is no doubting that it is a subtle effect and not one that most people could understand without a great deal of careful explanation which it is unlikely to get in the pages of a newspaper.

This entry was posted on August 30, 2009 at 12:03 pm and is filed under Bad Statistics with tags Medical Trials, placebo, Simpson's Paradox. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.