Frequentism: the art of probably answering the wrong question

Posted in Bad Statistics with tags , , , , , , on September 15, 2014 by telescoper

Popped into the office for a spot of lunch in between induction events and discovered that Jon Butterworth has posted an item on his Grauniad blog about how particle physicists use statistics, and the ‘5σ rule’ that is usually employed as a criterion for the detection of, e.g. a new particle. I couldn’t resist bashing out a quick reply, because I believe that actually the fundamental issue is not whether you choose 3σ or 5σ or 27σ but what these statistics mean or don’t mean.

As was the case with a Nature piece I blogged about some time ago, Jon’s article focuses on the p-value, a frequentist concept that corresponds to the probability of obtaining a value at least as large as that obtained for a test statistic under a particular null hypothesis. To give an example, the null hypothesis might be that two variates are uncorrelated; the test statistic might be the sample correlation coefficient r obtained from a set of bivariate data. If the data were uncorrelated then r would have a known probability distribution, and if the value measured from the sample were such that its numerical value would be exceeded with a probability of 0.05 then the p-value (or significance level) is 0.05. This is usually called a ‘2σ’ result because for Gaussian statistics a variable has a probability of 95% of lying within 2σ of the mean value.

Anyway, whatever the null hypothesis happens to be, you can see that the way a frequentist would proceed would be to calculate what the distribution of measurements would be if it were true. If the actual measurement is deemed to be unlikely (say that it is so high that only 1% of measurements would turn out that large under the null hypothesis) then you reject the null, in this case with a “level of significance” of 1%. If you don’t reject it then you tacitly accept it unless and until another experiment does persuade you to shift your allegiance.

But the p-value merely specifies the probability that you would reject the null-hypothesis if it were correct. This is what you would call making a Type I error. It says nothing at all about the probability that the null hypothesis is actually a correct description of the data. To make that sort of statement you would need to specify an alternative distribution, calculate the distribution based on it, and hence determine the statistical power of the test, i.e. the probability that you would actually reject the null hypothesis when it is incorrect. To fail to reject the null hypothesis when it’s actually incorrect is to make a Type II error.

If all this stuff about p-values, significance, power and Type I and Type II errors seems a bit bizarre, I think that’s because it is. It’s so bizarre, in fact, that I think most people who quote p-values have absolutely no idea what they really mean. Jon’s piece demonstrates that he does, so this is not meant as a personal criticism, but it is a pervasive problem that results quoted in such a way are intrinsically confusing.

The Nature story mentioned above argues that in fact that results quoted with a p-value of 0.05 turn out to be wrong about 25% of the time. There are a number of reasons why this could be the case, including that the p-value is being calculated incorrectly, perhaps because some assumption or other turns out not to be true; a widespread example is assuming that the variates concerned are normally distributed. Unquestioning application of off-the-shelf statistical methods in inappropriate situations is a serious problem in many disciplines, but is particularly prevalent in the social sciences when samples are typically rather small.

While I agree with the Nature piece that there’s a problem, I don’t agree with the suggestion that it can be solved simply by choosing stricter criteria, i.e. a p-value of 0.005 rather than 0.05 or, in the case of particle physics, a 5σ standard (which translates to about 0.000001!  While it is true that this would throw out a lot of flaky ‘two-sigma’ results, it doesn’t alter the basic problem which is that the frequentist approach to hypothesis testing is intrinsically confusing compared to the logically clearer Bayesian approach. In particular, most of the time the p-value is an answer to a question which is quite different from that which a scientist would actually want to ask, which is what the data have to say about the probability of a specific hypothesis being true or sometimes whether the data imply one hypothesis more strongly than another. I’ve banged on about Bayesian methods quite enough on this blog so I won’t repeat the arguments here, except that such approaches focus on the probability of a hypothesis being right given the data, rather than on properties that the data might have given the hypothesis.

I feel so strongly about this that if I had my way I’d ban p-values altogether…

Not that it’s always easy to implement a Bayesian approach. It’s especially difficult when the data are affected by complicated noise statistics and selection effects, and/or when it is difficult to formulate a hypothesis test rigorously because one does not have a clear alternative hypothesis in mind. Experimentalists (including experimental particle physicists) seem to prefer to accept the limitations of the frequentist approach than tackle the admittedly very challenging problems of going Bayesian. In fact in my experience it seems that those scientists who approach data from a theoretical perspective are almost exclusively Baysian, while those of an experimental or observational bent stick to their frequentist guns.

Coincidentally a paper on the arXiv not long ago discussed an interesting apparent paradox in hypothesis testing that arises in the context of high energy physics, which I thought I’d share here. Here is the abstract:

The Jeffreys-Lindley paradox displays how the use of a p-value (or number of standard deviations z) in a frequentist hypothesis test can lead to inferences that are radically different from those of a Bayesian hypothesis test in the form advocated by Harold Jeffreys in the 1930’s and common today. The setting is the test of a point null (such as the Standard Model of elementary particle physics) versus a composite alternative (such as the Standard Model plus a new force of nature with unknown strength). The p-value, as well as the ratio of the likelihood under the null to the maximized likelihood under the alternative, can both strongly disfavor the null, while the Bayesian posterior probability for the null can be arbitrarily large. The professional statistics literature has many impassioned comments on the paradox, yet there is no consensus either on its relevance to scientific communication or on the correct resolution. I believe that the paradox is quite relevant to frontier research in high energy physics, where the model assumptions can evidently be quite different from those in other sciences. This paper is an attempt to explain the situation to both physicists and statisticians, in hopes that further progress can be made.

This paradox isn’t a paradox at all; the different approaches give different answers because they ask different questions. Both could be right, but I firmly believe that one of them answers the wrong question.

35 Comments »

Parliamo Glasgow

Posted in Politics, Television with tags , , , on September 15, 2014 by telescoper

Whatever the outcome of Thursday’s referendum on independence, it’s clear that we who live South of the Border need to try harder to understand the Scots much better than we have so far. Because Glasgow and its environs appear to be hotbeds of the pro-secession vote, I can think of no better way to begin this process than by learning more about Glaswegian and the way what she is spoke, in order to improve mutual respect and foster dialogue.

Here are some useful lessons based on everyday situations and characters, as portrayed by the inimitable Stanley Baxter…

 

 

6 Comments »

The Arrival of the Queen of Sheba

Posted in Biographical, Music with tags , , , on September 14, 2014 by telescoper

It’s been such a hectic week getting ready for our new arrivals at the University of Sussex that I’ve been largely limited to posting short items and recycled material. Today is no exception either, as I have been on campus again for another Freshers’ induction week event and now have to prepare a talk for new students in the School of Mathematical and Physical Sciences for tomorrow morning. Anyway, all these busy preparations made me think today of the famous instrumental passage from Act III of Handel’s Oratorio Solomon which depict in wonderfully lively fashion  similar preparations preceding the Arrival of the Queen of Sheba. I know that’s a feeble pretext for posting a bit of music, but I thought I’d at least put a little twist on it by including a performance rather different from what you might be expecting.

This version of The Arrival of the Queen of Sheba is by wonderful Welsh harpist Glenda Clwyd whom I remember hearing play at a couple of events when I was living in Cardiff. This clip is of an impromptu performance and there are a couple of small mistakes, but I think it’s a lovely rendition, the uniquely gentle sound of the harp making it less frantic than most versions.

 

 

3 Comments »

Freshers’ Week Reminiscences

Posted in Biographical, Education with tags , , , , on September 13, 2014 by telescoper

So here I am again, on campus, on a Saturday, this time to attend some receptions for new students (“Freshers”) who have just arrived at the University of Sussex to start their courses. I always enjoy meeting the new intake at this time of year; we sometimes call them “The Autumn Collection”, although it’s only mid-September and definitely not autumn yet. In fact it’s very warm and sunny and summery on Falmer campus today. The  downside of these annual events is that the students look much younger every year, so every one makes me feel a lot older than the one before!

Looking through my back catalogue of blog posts I realize that this blog is six years old next week. One of my first blog posts was about  memories of my own first day at University and it seems appropriate to repeat some of it here. I notice actually that virtually all Freshers’ weeks I’ve written about over the past six years have been accompanied by fine weather. I find this kind of weather a bit spooky because it always takes me back to the time when I left home to go to University, as thousands of fledgling students are about to do this year in their turn. I did it 32 years ago, getting on a train at Newcastle Central station with my bags of books and clothes. I said goodbye to my parents there. There was never any question of them taking me in the car all the way to Cambridge. It wasn’t practical and I wouldn’t have wanted them to do it anyway. After changing from the Inter City at Peterborough onto a local train, we trundled through the flatness of East Anglia until it reached Cambridge. The weather, at least in my memory, was exactly like today.

I don’t remember much about the actual journey, but I must have felt a mixture of fear and excitement. Nobody in my family had ever been to University before, let alone to Cambridge. Come to think of it, nobody from my family has done so since either. I was a bit worried about whether the course I would take in Natural Sciences would turn out to be difficult, but I think my main concern was how I would fit in generally.

I had been working between leaving school and starting my undergraduate course, so I had some money in the bank and I was also to receive a full grant. I wasn’t really worried about cash. But I hadn’t come from a posh family and didn’t really know the form. I didn’t have much experience of life outside the North East either. I’d been to London only once before going to Cambridge, and had never been abroad.

I didn’t have any posh clothes, a deficiency I thought would mark me as an outsider. I had always been grateful for having to wear a school uniform (which was bought with vouchers from the Council) because it meant that I dressed the same as the other kids at School, most of whom came from much wealthier families. But this turned out not to matter at all. Regardless of their family background, students were generally a mixture of shabby and fashionable, like they are today. Physics students in particular didn’t even bother with the fashionable bit. Although I didn’t have a proper dinner jacket for the Matriculation Dinner, held for all the new undergraduates, nobody said anything about my dark suit which I was told would be acceptable as long as it was a “lounge suit” (whatever that is).

Taking a taxi from the station, I finally arrived at Magdalene College. I waited outside, a bundle of nerves, for some time before entering the Porter’s Lodge and starting my life as a student. My name was found and ticked off and a key issued for my room in the Lutyens building. It turned out to be a large room, with a kind of screen that could be pulled across to divide the room into two, although I never actually used this contraption. There was a single bed and a kind of cupboard containing a sink and a mirror in the bit that could be hidden by the screen. The rest of the room contained a sofa, a table, a desk, and various chairs, all of them quite old but solidly made. Outside my  room, on the landing, was the gyp room, a kind of small kitchen, where I was to make countless cups of tea over the following months, although I never actually cooked anything there.

I struggled in with my bags and sat on the bed. It wasn’t at all like I had imagined. I realized that no amount of imagining would ever really have prepared me for what was going to happen at University.

I  stared at my luggage. I suddenly felt like I had landed on a strange island where I didn’t know anyone, and couldn’t remember why I had gone there or what I was supposed to be doing. I’ve had that feeling ever since, but after 32 years I think I’m used to it.

4 Comments »

Say no to the commercialization of education!

Posted in The Universe and Stuff with tags , on September 12, 2014 by telescoper

There is much complaint these days about the alleged  commercialization of UK Higher Education, so I  wanted to take this opportunity to state Virgin Airlines that I will not be taking this as a Carling cue to introduce any form of commercial Coca Cola sponsorship of any Corby Trouser Press form into the School of Mathematical Macdonalds and Panasonic Physical Sciences, and certainly not into this Burger King blog.

This week I’ve been working hard preparing for the new Marks and Spencer term  and especially for the arrival of our new  Samsung students who will be starting their  Dixons degrees next week.  The Nokia preparations have gone pretty well  although I have had Betfair trouble cramming all the Sainsbury things I’ve had to do this BMW week, so I’ll be in Tesco tomorrow and Wonga Sunday to finish off a few Pizza Express jobs, but at least I’ll be able to attend the Vodafone Vice-Chancellor’s receptions for new students on the Carlsberg campus this Waitrose weekend.

In between these Ericsson events I hope to find some time to write a little more Morrisons of the second edition of my book on cosmology, including stuff about the Carphone Warhouse cosmic microwave background (CMB) which produces some of the noise on a Sony television screen, a  Classic FM signal from the edge of the Next Universe.  The CMB plays an Emirates important role in TK Maxx cosmology as it is the Marlboro smoking gun of the Sainsbury Big Bang and established our Standard Life model of the L’Oreal Universe. The old British Airways edition is a bit out of Aviva date so I will be updating it with Starbucks references to  the Planck First Direct results, although I obviously haven’t decided yet what to say about Barclays BICEP2.  I think I’ll be adding a Goodfella’s Pizza paragraph or two referring to the House of Fraser Hubble Crown Paints  Ultra Deep Kentucky Fried Chicken Field as well.

Anyway, for now its  Thank God It’s Friday time to go HSBC home and drink several Dorothy Perkins glasses of Amazon wine.

Comet Sale Now On!

 

2 Comments »

September Song – the Django Reinhardt version

Posted in Jazz with tags , , on September 11, 2014 by telescoper

Summer’s drawing to a close and preparing for the imminent arrival of new intake of students is taking up a lot of my time this week, so I thought I’d just put up something I’ve posted before, in the form of a piece of music that celebrates the genius of Django Reinhardt, the great Belgian-born gypsy guitarist who overcame the terrible  injuries he suffered as a child (in a fire in his caravan) to become one of the greatest jazz musicians of all time.  He had a unique style of playing the guitar he invented himself to get around the fact that the third and fourth digits on his left hand were so badly burned he could effectively only use two fingers. He also had an unparalleled gift for melodic improvisation that won him admirers all around the world and across all styles of music. Add him to your list of famous Belgians right away, for he was most certainly a musical genius.

Here he’s playing the beautifully poignant September Song, by Kurt Weill:

Oh, it’s a long, long while from May to December
But the days grow short when you reach September

 

1 Comment »

As Summer into Autumn slips

Posted in Poetry with tags , , , on September 10, 2014 by telescoper

As Summer into Autumn slips
And yet we sooner say
“The Summer” than “the Autumn,” lest
We turn the sun away,

And almost count it an Affront
The presence to concede
Of one however lovely, not
The one that we have loved —

So we evade the charge of Years
On one attempting shy
The Circumvention of the Shaft
Of Life’s Declivity.

by Emily Dickinson (1830-1886)

2 Comments »

Scotland Should Decide…

Posted in Bad Statistics, Politics, Science Politics with tags , , , , , , , , , on September 9, 2014 by telescoper

There being less than two weeks to go before the forthcoming referendum on Scottish independence, a subject on which I have so far refrained from commenting, I thought I would write something on it from the point of view of an English academic. I was finally persuaded to take the plunge because of incoming traffic to this blog from  pro-independence pieces here and here and a piece in Nature News on similar matters.

I’ll say at the outset that this is an issue for the Scots themselves to decide. I’m a believer in democracy and think that the wishes of the Scottish people as expressed through a referendum should be respected. I’m not qualified to express an opinion on the wider financial and political implications so I’ll just comment on the implications for science research, which is directly relevant to at least some of the readers of this blog. What would happen to UK research if Scotland were to vote yes?

Before going on I’ll just point out that the latest opinion poll by Yougov puts the “Yes” (i.e. pro-independence) vote ahead of “No” at 51%-49%. As the sample size for this survey was only just over a thousand, it has a margin of error of ±3%. On that basis I’d call the race neck-and-neck to within the resolution of the survey statistics. It does annoy me that pollsters never bother to state their margin of error in press released. Nevertheless, the current picture is a lot closer than it looked just a month ago, which is interesting in itself, as it is not clear to me as an outsider why it has changed so dramatically and so quickly.

Anyway, according to a Guardian piece not long ago.

Scientists and academics in Scotland would lose access to billions of pounds in grants and the UK’s world-leading research programmes if it became independent, the Westminster government has warned.

David Willetts, the UK science minister, said Scottish universities were “thriving” because of the UK’s generous and highly integrated system for funding scientific research, winning far more funding per head than the UK average.

Unveiling a new UK government paper on the impact of independence on scientific research, Willetts said that despite its size the UK was second only to the United States for the quality of its research.

“We do great things as a single, integrated system and a single integrated brings with it great strengths,” he said.

Overall spending on scientific research and development in Scottish universities from government, charitable and industry sources was more than £950m in 2011, giving a per capita spend of £180 compared to just £112 per head across the UK as a whole.

It is indeed notable that Scottish universities outperform those in the rest of the United Kingdom when it comes to research, but it always struck me that using this as an argument against independence is difficult to sustain. In fact it’s rather similar to the argument that the UK does well out of European funding schemes so that is a good argument for remaining in the European Union. The point is that, whether or not a given country benefits from the funding system, it still has to do so by following an agenda that isn’t necessarily its own. Scotland benefits from UK Research Council funding, but their priorities are set by the Westminster government, just as the European Research Council sets (sometimes rather bizarre) policies for its schemes. Who’s to say that Scotland wouldn’t do even better than it does currently by taking control of its own research funding rather than forcing its institutions to pander to Whitehall?

It’s also interesting to look at the flipside of this argument. If Scotland were to become independent, would the “billions” of research funding it would lose (according to the statement by Willetts, who is no longer the Minister in charge) benefit science in what’s left of the United Kingdom? There are many in England and Wales who think the existing research budget is already spread far too thinly and who would welcome an increase south of the border. If this did happen you could argue that, from a very narrow perspective, Scottish independence would be good for science in the rest of what is now the United Kingdom, but that depends on how much the Westminster government sets the science budget.

This all depends on how research funding would be redistributed if and when Scotland secedes from the Union, which could be done in various ways. The simplest would be for Scotland to withdraw from RCUK entirely. Because of the greater effectiveness of Scottish universities at winning funding compared to the rest of the UK, Scotland would have to spend more per capita to maintain its current level of resource, which is why many Scottish academics will be voting “no”. On the other hand, it has been suggested (by the “yes” campaign) that Scotland could buy back from its own revenue into RCUK at the current effective per capita rate  and thus maintain its present infrastructure and research expenditure at no extra cost. This, to me, sounds like wanting to have your cake and eat it,  and it’s by no means obvious that Westminster could or should agree to such a deal. All the soundings I have taken suggest that an independent Scotland should expect no such generosity, and will get actually zilch from the RCUK.

If full separation is the way head, science in Scotland would be heading into uncharted waters. Among the questions that would need to be answered are:

  •  what will happen to RCUK funded facilities and staff currently situated in Scotland, such as those at the UKATC?
  •  would Scottish researchers lose access to facilities located in England, Wales or Northern Ireland?
  •  would Scotland have to pay its own subscriptions to CERN, ESA and ESO?

These are complicated issues to resolve and there’s no question that a lengthy process of negotiation would be needed to resolved them. In the meantime, why should RCUK risk investing further funds in programmes and facilities that may end up outside the UK (or what remains of it)? This is a recipe for planning blight on an enormous scale.

And then there’s the issue of EU membership. Would Scotland be allowed to join the EU immediately on independence? If not, what would happen to EU funded research?

I’m not saying these things will necessarily work out badly in the long run for Scotland, but they are certainly questions I’d want to have answered before I were convinced to vote “yes”. I don’t have a vote so my opinion shouldn’t count for very much, but I wonder if there are any readers of this blog from across the Border who feel like expressing an opinion?

 

11 Comments »

A Brain Teaser from SWALEC

Posted in Uncategorized with tags , , on September 8, 2014 by telescoper

As I wend my way Brightonwards after a week in Wales, I thought I’d request your assistance in deciphering part of a letter I received from SWALEC, the company that supplies gas and electricity to my Cardiff residence.

If anyone out there can explain the logic that leads from a credit of £163.54 to an increase in my monthly payment – other than the obvious one that SWALEC are trying it on – I’d be most grateful!

3 Comments »

Laniakea – Our Local Supercluster

Posted in The Universe and Stuff with tags , , , on September 6, 2014 by telescoper

My week of self-imposed isolation is almost over so I suppose I should try to re-acclimatize myself to the world (or at least the world of the internet)  by doing a quick post of a nice video. I remember Brent Tully talking at the conference I went to in Estonia earlier this summer about the work he has been doing with his collaborators on using the local peculiar velocity field to map structures in the galaxy distribution. Now the paper is out in the journal Nature. Laniakea is the name the group chose for the local supercluster which has been known about for some time, but this work provides a more detailed map. The name Laniakea means “immeasurable heaven” in Hawaiian, from “lani” for ‘heaven’ and “akea” for ‘spacious’ or ‘immeasurable’.  Rather disappointingly it has nothing to do with Ikea, so sheds little light on my own theory of the Universe.

 

1 Comment »

« Older Entries
Newer Entries »