Archive for Jon Butterworth

Cuts, Commitments and Contradictions – guest post by Lucien Heurtier

Posted in Science Politics with tags , , , , , , , on March 9, 2026 by telescoper

Lucien Heurtier is a Postdoctoral Research Associate at King’s College London in the group of Theoretical Particle Physics & Cosmology. He contacted me yesterday to ask if I would use this platform to share the a blog post he wrote about the events at last week’s Select Commitee meetings about the crisis at the Science and Technology Facilities, in order to boost its circulation. I am happy to do so. I have changed the formatting a little, but not any of the content.

–0–

Over the past week, three key meetings brought together members of the Particle Physics, Astronomy, and Nuclear Physics (PPAN) community with ministers, Members of Parliament, and representatives of UKRI and STFC. For the PPAN community, these discussions were particularly significant. They not only shed light on some of the underlying causes of the current financial pressures facing the programme, but also revealed what appears to be a growing disconnect between the strategic priorities emerging within UKRI and the concerns expressed by government, STFC leadership, and the PPAN research community itself.

In this article, I attempt to capture how researchers across the PPAN community have interpreted and reacted to these meetings. I discuss how this perceived disconnect relates to the developments of the past several months, and what these events may mean for what comes next.

The House of Lords Acknowledges a ‘Very Particular Problem Around STFC’

On Tuesday, 3rd of March, Rt Hon Liz Kendall MP (Secretary of State at DSIT), Lord Patrick Vallance (Minister of State at DSIT), and Emran Mian (Permanent Secretary at DSIT) appeared before the House of Lords Science, Innovation and Technology Committee, which questioned them on the UKRI funding strategy and its impact on PPAN science.

From the start, Lord Mair (the chair) questioned the Minister: “As you are probably aware, several research councils have paused grants and announced cuts to basic science funding”, he said. “Is it the Government’s policy to cut funding for curiosity-driven research—from bucket 1—in favour of research for the other two buckets?”. Lord Vallance of Balham responded that “There have been no cuts in basic, curiosity-driven research”, although he admitted that “there is a very particular problem around STFC, […] but it is not the case that there have been cuts in any of the other areas”. So the stage is set: STFC is the only council facing explicit cuts. This might sound like a technicality to some, but for the PPAN community, simply getting the minister to acknowledge that STFC is facing budget cuts is already a success.

Among others, an important question comes up: “Would it be right to say that QR funding is being assumed to principally support bucket 1?” Indeed, in recent communications, UKRI has repeatedly classified QR research (QR standing for Quality Related) as being entirely part of the budget for bucket-1. In fact, it represents roughly 60% of the total budget in that bucket. However, Lord Vallance confessed, “No, it is going to support whatever the universities want it to support.” He even explicitly said that “that may be reallocated to other buckets, actually”. This obviously raises the question of whether curiosity-driven research is actually protected, as the government and UKRI have been repeating for months, and why QR research was entirely counted as contributing to bucket 1. Yet Lord Vallance simply said that “Sir Ian Chapman and the team—I think correctly—decided that trying to divide QR up in a complicated formula was bureaucratic”. Make of that what you will.

Lord Vallance then acknowledged poor communications from UKRI: “We can all agree that has not been done well”, he said. He then brought up the STFC case himself: “STFC is unusual in research councils because it has a very large infrastructure pot, and it also funds particle physics and astronomy”. “There is something that needs to be resolved there”, he repeated, “the basic, curiosity-driven, investigator-led research in that bucket needs to be protected”. Once again, such a statement is extremely important for PPAN. The minister is acknowledging that, beyond bad communication from UKRI, there is a problem here, and that cuts in STFC research are not in line with the idea that curiosity-driven research is protected, which Lord Vallance clearly appears to care about.

The committee kept asking: “We are hearing that there is a 30% reduction—the budget itself has not changed, but there is a shifting in the budget for STFC”, said the Baroness Willis of Summertown. “The ringfencing for the blue skies [Drayson partition] has gone from that structure. Is that understanding correct?” “No”, said Lord Vallance, “there was no hard partition in that. It has always been tensioned against the two things”. “The international spend has gone up by about 20% at a time when domestic spend has gone up at about 11% over a period of six or seven years. That has put big pressure on the overall system”, he said. “In previous years, the overspend in STFC has then been absorbed by the other research councils, so there has been a strange picture where other research councils have actually ended up having to give money into the system to cover that. We need to fix that. We need a sustainable, proper, well thought-through, structured way to fund the infrastructure. I am very determined that UKRI must find a way to look after so-called PPAN—particle physics and astronomy.” This statement, I think, kept many of my colleagues in suspense before finally prompting a collective sigh of relief.

Later during the meeting, Lord Drayson insisted: “This is not a new problem”, he said, “We saw this back in the financial crash of 2007-08. That is when we put in those protections to ensure that the other budgets were not hit.” “The Government needs to be able to recognise the long-term funding requirements for the science budget to protect these facilities”, he added. To which Liz Kendall responded that “We are here again, but our commitment to long-term funding of these areas is absolutely there”. This very much sounded as though DSIT is determined to protect PPAN science, but also facilities, against their potential cost increase. We will hold them to their word.

The minister was then extensively questioned about the new ‘bucket’ framework. “you will accept, I think, that the reorganisation that UKRI is bringing in—you have mentioned its looking to facilitate the removal of duplication and have cross-cutting thematic research—means that the complexity of the decision-making process is becoming more opaque”, said Lord Drayson. “I worry that by insisting that this over here is blue sky and this over here is applied, you risk leaving out or not concentrating enough on the most interesting things”, said Lord Stern of Brentford. “It is absolutely one of the risks”, responded Vallance, adding that UKRI “will look at how to make that work across buckets, and it is going to put in systems”. Unfortunately, nothing more concrete than that emerged from the meeting. But Lord Vallance made it very clear, “We view the first bucket as protecting that against what I have seen in companies and see as a risk in government, which is somebody looking at the £14.5 billion and saying ‘Well, it wouldn’t really matter if we didn’t do that for a while’. It matters enormously because once you lose that, you lose it for a very long time, and it is that work that ultimately creates wealth in 10 or 20 or 30 years’ time, even though I cannot tell you which bits of it are going to create wealth.” Again, such a commitment that the government is going to protect blue skies science is essential for PPAN.

Many other important things were raised during the rest of this hearing, but this part is what mattered the most to the PPAN community. As we will see later, the notion that curiosity-driven and PPAN science must be protected clearly contrasts with a very different attitude from UKRI…

PPAN Early Career Researchers and Advanced Fellows Raise Concerns with STFC and UKRI — Only to Be Dismissed

The same day, a delegation of early-career researchers (postdoctoral researchers and PhD students) and advanced fellows (holding advanced fellowships such as the Ernest Rutherford, Future Leader, or Royal Society Research Fellowships) from all components of the PPAN community were invited for a ‘consultation’ meeting with Sir Ian Chapman (CEO of UKRI), Prof. Michele Dougherty (head of STFC), and Prof. Graham Blair (STFC Executive Director of Programmes), accompanied by an external observer from the Institute of Physics, Elizabeth Chamberlain. “I would be happy to meet with you to discuss the situation so that we can explain the details and discuss your suggestions”, Sir Ian Chapman wrote in his invitation two weeks before the meeting.

We came prepared. We gathered a team of representatives, with people from all PPAN areas of research and various career stages. We sent the CEO of UKRI a list of questions a week before the meeting so that our suggestions could better reflect the realities on the ground. Our questions were ignored. UKRI is certainly busy these days. We therefore refined our arguments and developed proposals that, in our view, represented the minimum needed to support our community.

Yet we ran into a wall. To be fair, the meeting format allowed an open discussion, in which both sides could clearly express their ideas, which we were particularly grateful for. But what emerged from the meeting was a profound disconnect between the alarms raised by the PPAN community—based on scientific excellence and sovereignty over key research capabilities and highly-qualified scientists—and the arguments advanced by both UKRI and STFC representatives, exclusively based on accounting cost-reduction arguments.

“You know why we’re here. 30% cuts.” began Dr Kirsty Duffy. But that’s not how they see it. Indeed, from the UKRI perspective, STFC must have a flat budget, as all other councils do, and if STFC costs increase, it must accept corresponding cuts to its grant funding. It is as simple as that, and at no point during the meeting did either Sir Ian Chapman or Michele Dougherty consider a different possibility. From the PPAN point of view, things are really different: “not only the expected cuts, but the current delay has already removed a cohort of ECRs”, said Dr Simon Williams. “Rebuilding is not a matter of returning money or not making a similar cut”, he said, “has the effect been forecast on the output of the community?” “I don’t know”, admits Sir Ian Chapman. “We have a budget, and we have to work within it. It’s where it is from where we are”, repeats Prof Dougherty. And that was it.

ECRs have asked repeatedly for details of the cost overruns and where they come from—this was part of the formal request for information sent before the meeting, and multiple requests for that information during the meeting. Unfortunately, this information has not been provided, and ECRs expressed that this leads the community to feel it is a deliberate decision by UKRI to cut PPAN in favour of facilities, particularly given that the overall STFC budget will be flat. Sir Ian Chapman said that the main driver of cost pressures was starting too many projects, and that energy costs were a small fraction (which appears to contradict previous public statements). Prof Dougherty said “the majority of the cut is within STFC, where the vast majority of the increase in costs comes”, although Sir Ian Chapman said that no decision on how cost savings would be apportioned between PPAN and STFC facilities had been made yet.

Probably the only positive outcome of this meeting: Prof Dougherty clarified that a 30% cut is the “worst case scenario” and that the Science Board has been asked to put together scenarios for 10%, 20%, and 30% cuts. She clarified that this was relative to the fiscal year 2024 budget, and that the PPAN grants have already been cut 15% compared to that. So perhaps we should have considered ourselves fortunate, as a 10% scenario would mean the grant line will be going up again, slightly… Michele Dougherty said she will take those scenarios to UKRI and the Science Minister before they reach a final decision.

Advanced fellows made the case that existing cuts have already hurt the astronomy community very badly: “The funding gap in departments had the direct effect that people can no longer be named on grants”, said Laura Wolz. “People going abroad, not finding other positions, those are real effects with real consequences”. “The leadership we have internationally will be undermined if funding changes overnight”, added Dr Harriett Watson. “Any ECR in this room wants to be an international leader, but the pipeline is cut short if we remove funding”, she said. The least we would have hoped for is for UKRI to listen to the concerns, acknowledge that it is critical and formulate the intention to bring that problem to the government in one way or the other to attempt to solve it. The reaction we encountered, however, was rather less encouraging. “Do you accept that this is happening now?” insisted Dr Williams, “the effects of those cuts and delays are already leading to losing a generation of ECRs, who are leaving outside of the UK and won’t come back”, he said. “Yes, I grasp we will lose some postdocs as a result. I hope we don’t lose all. I can’t see a scenario where we would sign on consolidated grants that only cover academic staff time.” A comforting thought for ECRs: they might not be completely wiped out after all… “Perhaps some crumbs of comfort”, adds Sir Ian Chapman. “In a previous job, we had to implement a 30% budget cut. For three years, we had no PhD students and no postdocs, and we had to make compulsory redundancies among staff. It was a bleak period, and everything was under challenge. But today that community is in rude health, and its budget has been growing year on year.” The message is clear. We need to accept that PPAN will be hurt to unprecedented levels, but to look at the bright side: Time heals all wounds.

We also raised the issue of the Infrastructure Fund in light of the cancellation of some PPAN projects, in particular the LHCb upgrade. Both STFC and UKRI stressed that projects in other councils were also cut, but the nature of the damage to our international reputation was raised. Sir Ian Chapman repeated that the funding had not been awarded, but we insisted that funding had been allocated with the award subject to business case approval, for which UKRI had not read the business case. Sir Ian told us that all funding was subject to spending review and that tough decisions needed to be made. Prof. Dougherty noted that she recused herself from the Investment Advisory Committee’s decision-making process.

One “upside” that UKRI is always keen to remind the community is that PPAN research might be able to access funds from other buckets, through, for instance, AI and quantum-oriented projects. An upside that, Ian Chapman admits, “is not accessible yet”. “Is it dangerous to cut PPAN, which is more blue sky and where much of quantum and AI came from, for something that gives growth now but maybe not sovereignty in the future?” asked Dr Simon Williams. “Complicated answer”, says Chapman, “not all within our gift”, he confesses.

And this is something we are all afraid of in PPAN, including for physicists who are experts in machine learning but whose purpose is entirely curiosity-driven. So I asked the CEO of UKRI, “People working on AI within the PPAN community are actually afraid that they may not be able to access other buckets that easily. Will part of the budget dedicated to AI actually be guaranteed to be accessible to PPAN research?” “Well,” said Sir Ian, “it will be open to everybody, and accessible to you, but money will go to highest-impact applications…”. The idea of partitioning the budget from other buckets so a fraction of it is guaranteed to go to PPAN science is not on the table, Ian Chapman confirmed to me after the meeting, as the idea of the buckets is to get rid of “disciplinary rigidity”. In other words, the amount of funding accessible to PPAN from other buckets cannot be quantified.

The idea of UKRI providing STFC with more money from councils that have decreasing cost forecasts is also not an option: “In previous years, STFC has gone overboard, and others compensated […] Imagine being in medical, how would you feel about this?” answers Chapman. I thus asked, “If it is the case that UKRI doesn’t have enough money to rescue PPAN research, then should UKRI not ask the government for more money specifically for STFC, so UKRI doesn’t have to sacrifice an entire field of research?” “We do that every day of every year”, says Chapman. One would hope so.

In short, none of our concerns can be reasonably addressed; the blame is on past decisions from STFC and UKRI, and the best UKRI and STFC can do now is to optimise the way they will implement cuts, through an exercise of reprioritisation. As representatives of the PPAN community in this meeting, needless to say that these conclusions were far from satisfactory.

The SIT Select Committee Rescues PPAN from STFC ‘Cutting Its Tree by the Roots’

The following day, on Wednesday 4th of March, two panels were heard by the Science, Innovation, and Technology select committee, in the House of Commons. Prof Jon Butterworth, Prof Catherine Heymans (Royal Astronomer of Scotland), and Dr Simon Williams represented the PPAN community and explained to the committee why the expected 30% cuts to PPAN grant funding announced by STFC and UKRI would be devastating for the country. After that, Prof Michele Dougherty, head of STFC and the Royal Astronomer of England, explained to the committee why she considers such cuts necessary, despite UKRI as a whole seeing its budget increase.

The first panel made very clear statements regarding the importance of PPAN science and how devastating a 30% cut would be for all the existing programmes and our international reputation. Prof Heymans started by listing the many international astronomy projects that are at risk because of these cuts. “The Vera Rubin Observatory is the biggest camera in the world, we have started making a movie of the universe” she said, and “this sort of cut means we will not be able to process that data”. Prof Butterworth reminded the committee that the LHC is “the most powerful microscope we’ve ever built”, and highlighted how essential LHCb is “to scrutinise the origins of our universe”. “Without it”, he warned, “we may end up missing some very key data there”. Prof Heymans added, “This is what gets people into physics to study at university, but then they go out and do all the amazing things. To cut these blue-skies areas of research, which are the gateway for these very important areas for the growth of our country, this is really not what the UK should be doing right now”. Freddie van Mierlo MP asked, “Does this impact how we are seen internationally?” Prof Butterworth did not hesitate to answer: “Very much”. Dame Chi Onwurah MP then asked “if funding was available in two years, would we be able to get back in?” Butterworth answered that we would try but “we would certainly not be leading anymore”.

Dr Williams then stressed how critical these cuts would be (and already are) for hiring early-career researchers, such as postdocs and PhD students. “ECRs tend to be where the economic growth comes from”, he said, “cutting at this level would be catastrophic for UK science, very much like killing the tree by cutting the roots: you might not notice it for a while, but time will come when you do”. Dr Lauren Sullivan MP asked whether it would be beneficial for ECRs if a transition mechanism, for instance, funding extensions, were provided to ensure that the workforce is not lost while the funding framework is being changed. “I agree”, said Dr Williams, “the consultation should have been done before the change. The uncertainty that has been injected into the system is catastrophic.”

After these concerns were raised, the committee questioned Prof Dougherty, who mostly blamed the previous governance of UKRI and STFC, invoking “an overabundance of ambition” leading to a “difficult shortfall” she had to handle in the best way possible. This was not, she said, “what I signed up for”. She added, “All I can talk to is what I’ve been dealing with since I arrived”. Regarding the UK’s international reputation, she sadly accepted, “it does weaken our standing, certainly”.

Michele Dougherty also insisted that for UKRI to find a quick solution to the problem, “we need to share with UKRI what the impact of these cuts is, then a final decision can be made”. “Ian Chapman is very well aware that the community […] hoping that he will see what the impact is and whether there is a way to mitigate that impact, but I cannot speak for him”, she said.

Nonetheless, Martin Wrigley MP insisted, “we heard the budget of UKRI is increasing, so they are losing, who is winning?” Prof Dougherty said, “I do not have responsibility for these new buckets”. Martin Wrigley MP is therefore not convinced: “it sounds to me like you need to be more creative in your allocation of your expanding budget to your existing people rather than projects.”, but Dougherty answered she is not responsible for the way money can be accessed from other buckets for AI and quantum, and the only thing she can do is to tell her community that “there is real potential there”, which Wrigley considered “too passive in accepting what you’re being given”.

“There are other things that could be done”, says the Rt Hon Kit Malthouse MP, “as for example, reclassifying subscriptions that you pay as international treaty obligations”. “I am having that conversation with Ian Chapman, and with DSIT as well”, says Dougherty. But Rt Hon Kit Malthouse MP insisted that “UKRI’s budget over the years has been sort of manipulated to ensure that the DSIT budget is fully spent […] There is flexibility in there, so if you are having that conversation and it is resulting in 30% cuts for some of these, should we be saying to the Minister next time we get them in front of us, ‘Why did you say no to Professor Dougherty?’“. “Nobody has said no yet” stresses the head of STFC, “but I have been asked to look at the impact that the 30% will have. I need to follow through on that while I am having this conversation.”

“The extent of the impact on our existing science, scientists and early-career researchers is unacceptable”, concludes Dame Chi Onwurah, “can you give a commitment that you will look into bringing funding to close that gap on the short and medium term?”. “Yes” said Prof Dougherty. Needless to say the PPAN community now looks forward to seeing these words put into action.

Taken together, these meetings reveal a striking contrast. On the one hand, ministers and parliamentarians appear increasingly aware that the current trajectory risks serious damage to UK particle physics and astronomy. On the other hand, UKRI and STFC leadership insist that the constraints of the current funding framework leave them little room for manoeuvre. The result is a situation in which the problem is widely acknowledged, but its resolution remains uncertain. Ideas were proposed during these meetings that are all worth exploring, but will certainly require seeking further approval from the government. Dr Dougherty has committed to find short term solutions to mitigate the damage currently inflicted on the PPAN community, but it is unclear how.

The coming months will therefore be decisive in determining whether these warnings translate into concrete action, or whether the UK will accept the long-term consequences of cutting one of its most internationally successful scientific communities.

Dr Lucien Heurtier

London, 07/03/2026

Big Science is not the Problem – it’s Top-Down Management of Research

Posted in Finance, Science Politics, The Universe and Stuff with tags , , , , on June 2, 2015 by telescoper

I’m very late to this because I was away at the weekend, but I couldn’t resist making a comment on a piece that appeared in the Grauniad last week entitled How can we stop big science hovering up all the research funding? That piece argues for a new system of allocating research funding to avoid all the available cash being swallowed by a few big projects. This is an argument that’s been rehearsed many times before in the context of physics and astronomy, the costs of the UK contribution to facilities such as CERN (home of the Large Hadron Collider) and the European Southern Observatory being major parts of the budget of the Science and Technology Facilities Council that often threaten to squeeze the funds available for “exploiting” these facilities – in other words for doing science. What’s different about the Guardian article however is that it focusses on genomics, which has only recently threatened to become a Big Science.

Anyway, Jon Butterworth has responded with a nice piece of his own (also in the Guardian) with which I agree quite strongly. I would however like to make a couple of comments.

First of all, I think there are two different usages of the phrase “Big Science” and we should be careful not to conflate them. The first, which particularly applies in astronomy and particle physics, is that the only way to do research in these subjects is with enormous and generally very expensive pieces of kit. For this reason, and in order to share the cost in a reasonable manner, these fields tend to be dominated by large international collaborations. While it is indeed true that the Large Hadron Collider has cost a lot of money, that money has been spent by a large number of countries over a very long time. Moreover, particle physicists argued for that way of working and collectively made it a reality. The same thing happens in astronomy: the next generation of large telescopes are all transnational affairs.

The other side of the “Big Science” coin is quite a different thing. It relates to attempts to impose a top-down organization on science when that has nothing to do with the needs of the scientific research. In other words, making scientists in big research centres when it doesn’t need to be done like that. Here I am much more sceptical of the value. All the evidence from, e.g., the Research Excellence Framework is that there is a huge amount of top-class research going on in small groups here and there, much of it extremely innovative and imaginative. It’s very hard to justify concentrating everything in huge centres that are only Big because they’ve taken killed everything that’s Small, by concentrating resources to satisfy some management fixation rather than based on the quality of the research being done. I have seen far too many attempts by funding councils, especially the Engineering and Physical Sciences Research Council, to direct funding from the top down which, in most cases, is simply not the best way to deliver compelling science. Directed programmes rarely deliver exciting science, partly because the people directing them are not the people who actually know most about the field.

I am a fan of the first kind of Big Science, and not only for scientific reasons. I like the way it encourages us to think beyond the petty limitations of national politics, which is something that humanity desparately needs to get used to. But while Big Science can be good, forcing other science to work in Big institutes won’t necessarily make it better. In fact it could have the opposite effect, stifling the innovative approaches so often found in small groups. Small can be beautiful too.

Finally, I’d have to say that I found the Guardian article that started this piece of to be a bit mean-spirited. Scientists should be standing together not just to defend but to advance scientific research across all the disciplines rather than trying to set different kinds of researchers against each other. I feel the same way about funding the arts, actually. I’m all for more science funding, but don’t want to see the arts to be killed off to pay for it.

End of Term Balls

Posted in Biographical, Education with tags , , , , on April 18, 2015 by telescoper

I haven’t had time to post for the last couple of days because I’ve been too bust with end-of-term business (and pleasure). Yesterday (Friday) was the last day of teaching term and this week I had to get a lot of things finished because of various deadlines, as well as attending numerous meetings. It’s been quite an exhausting week, not just because of that but also because by tradition the two departments within the School of Mathematical and Physical Sciences at the University of Sussex, the Department of Mathematics and the Department of Physics & Astronomy, hold their annual Staff-Student Balls on consecutive days. When I arrived here just over two years ago I decided that I should attend both or neither, as to attend at only one would look like favouritism. In fact this is the third time I’ve attended both of them. Let no-one say I don’t take my obligations seriously.  It’s a tough job, but someone has to do it. Holding both balls so close together  poses some problems for a person of my age, but I coped and also tried to weigh them up relative to each other and see  which was  most impressive.

Actually, both were really well organized. The Mathematics Ball was held in the elegant Hilton Metropole hotel and the Physics one in the Holiday Inn, both on the seafront. As has been the case in previous years the Mathematics ball is a bit more refined and sedate, the Physics one a little more raucous. Also this year there was a very large difference in the number of people going, with over 200 at the Physics Ball and only just over half that number at the Mathematics one. In terms of all-round fun I have to declare the Physics Ball the winner last year, but both occasions were very enjoyable. I’d like to say a very public thank you to the organizers of both events, especially Sinem and Jordan for Mathematics and Francis for Physics. Very well done.

The highlight of the Physics Ball was an after-dinner speech by particle physicist Jon Butterworth, who has an excellent blog called Life and Physics on the Guardian website. I’ve actually been in contact with Jon many times through social media (especially Twitter) over a period of over six years, but we never actually met in person until last night. I think he was a bit nervous beforehand because he had never done an after-dinner speech before, in the end though his talk was funny and wise, and extremely well received. Mind you, I did make it easy for him by giving a short speech to introduce him, and after a speech by me almost anyone would look good!

Thereafter the evening continued with drinking and dancing. After a while most people present were rather tired and emotional.  I even think some might even have been drunk. I eventually got home about 2am, after declining an invitation to go to the after-party. I’m far too old for that sort of thing. Social events like this can be a little bit difficult, for a number of reasons. One is that there’s an inevitable “distance” between students and staff, not just in terms of age but also in the sense that the staff have positions of responsibility for the students. Students are not children, of course, so we’re not legally  in loco parentis, but something of that kind of relationship is definitely there. Although it doesn’t stop either side letting their hair down once in a while, I always find there’s a little bit of tension especially if the revels get a bit out of hand. To help occasions like this run smoothly I think it’s the responsibility of the staff members present to drink heavily in order to put the students at ease. United by a common bond of inebriation, the staff-student divide crumbles and a good time is had by all.

There’s another thing I find a bit strange. Chatting to students last night was the first time I had spoken to many of my students like that, i.e. outside the lecture  or tutorial. I see the same faces in my lectures day in, day out but all I do is talk to them about physics. I really don’t know much about them at all. But it is especially nice when on occasions like this students come up, as several did last night, and say that they enjoyed my lectures. Actually, it’s more than just nice. Amid all the bureaucracy and committee meetings, it’s very valuable to be reminded what the job is really all about.

 

P.S. Apologies for not having any pictures. I left my phone in the office on Friday when I went home to get changed. I will post some if anyone can supply appropriate images. Or, better still, inappropriate ones!

 

 

Frequentism: the art of probably answering the wrong question

Posted in Bad Statistics with tags , , , , , , on September 15, 2014 by telescoper

Popped into the office for a spot of lunch in between induction events and discovered that Jon Butterworth has posted an item on his Grauniad blog about how particle physicists use statistics, and the ‘5σ rule’ that is usually employed as a criterion for the detection of, e.g. a new particle. I couldn’t resist bashing out a quick reply, because I believe that actually the fundamental issue is not whether you choose 3σ or 5σ or 27σ but what these statistics mean or don’t mean.

As was the case with a Nature piece I blogged about some time ago, Jon’s article focuses on the p-value, a frequentist concept that corresponds to the probability of obtaining a value at least as large as that obtained for a test statistic under a particular null hypothesis. To give an example, the null hypothesis might be that two variates are uncorrelated; the test statistic might be the sample correlation coefficient r obtained from a set of bivariate data. If the data were uncorrelated then r would have a known probability distribution, and if the value measured from the sample were such that its numerical value would be exceeded with a probability of 0.05 then the p-value (or significance level) is 0.05. This is usually called a ‘2σ’ result because for Gaussian statistics a variable has a probability of 95% of lying within 2σ of the mean value.

Anyway, whatever the null hypothesis happens to be, you can see that the way a frequentist would proceed would be to calculate what the distribution of measurements would be if it were true. If the actual measurement is deemed to be unlikely (say that it is so high that only 1% of measurements would turn out that large under the null hypothesis) then you reject the null, in this case with a “level of significance” of 1%. If you don’t reject it then you tacitly accept it unless and until another experiment does persuade you to shift your allegiance.

But the p-value merely specifies the probability that you would reject the null-hypothesis if it were correct. This is what you would call making a Type I error. It says nothing at all about the probability that the null hypothesis is actually a correct description of the data. To make that sort of statement you would need to specify an alternative distribution, calculate the distribution based on it, and hence determine the statistical power of the test, i.e. the probability that you would actually reject the null hypothesis when it is incorrect. To fail to reject the null hypothesis when it’s actually incorrect is to make a Type II error.

If all this stuff about p-values, significance, power and Type I and Type II errors seems a bit bizarre, I think that’s because it is. It’s so bizarre, in fact, that I think most people who quote p-values have absolutely no idea what they really mean. Jon’s piece demonstrates that he does, so this is not meant as a personal criticism, but it is a pervasive problem that results quoted in such a way are intrinsically confusing.

The Nature story mentioned above argues that in fact that results quoted with a p-value of 0.05 turn out to be wrong about 25% of the time. There are a number of reasons why this could be the case, including that the p-value is being calculated incorrectly, perhaps because some assumption or other turns out not to be true; a widespread example is assuming that the variates concerned are normally distributed. Unquestioning application of off-the-shelf statistical methods in inappropriate situations is a serious problem in many disciplines, but is particularly prevalent in the social sciences when samples are typically rather small.

While I agree with the Nature piece that there’s a problem, I don’t agree with the suggestion that it can be solved simply by choosing stricter criteria, i.e. a p-value of 0.005 rather than 0.05 or, in the case of particle physics, a 5σ standard (which translates to about 0.000001!  While it is true that this would throw out a lot of flaky ‘two-sigma’ results, it doesn’t alter the basic problem which is that the frequentist approach to hypothesis testing is intrinsically confusing compared to the logically clearer Bayesian approach. In particular, most of the time the p-value is an answer to a question which is quite different from that which a scientist would actually want to ask, which is what the data have to say about the probability of a specific hypothesis being true or sometimes whether the data imply one hypothesis more strongly than another. I’ve banged on about Bayesian methods quite enough on this blog so I won’t repeat the arguments here, except that such approaches focus on the probability of a hypothesis being right given the data, rather than on properties that the data might have given the hypothesis.

I feel so strongly about this that if I had my way I’d ban p-values altogether…

Not that it’s always easy to implement a Bayesian approach. It’s especially difficult when the data are affected by complicated noise statistics and selection effects, and/or when it is difficult to formulate a hypothesis test rigorously because one does not have a clear alternative hypothesis in mind. Experimentalists (including experimental particle physicists) seem to prefer to accept the limitations of the frequentist approach than tackle the admittedly very challenging problems of going Bayesian. In fact in my experience it seems that those scientists who approach data from a theoretical perspective are almost exclusively Baysian, while those of an experimental or observational bent stick to their frequentist guns.

Coincidentally a paper on the arXiv not long ago discussed an interesting apparent paradox in hypothesis testing that arises in the context of high energy physics, which I thought I’d share here. Here is the abstract:

The Jeffreys-Lindley paradox displays how the use of a p-value (or number of standard deviations z) in a frequentist hypothesis test can lead to inferences that are radically different from those of a Bayesian hypothesis test in the form advocated by Harold Jeffreys in the 1930’s and common today. The setting is the test of a point null (such as the Standard Model of elementary particle physics) versus a composite alternative (such as the Standard Model plus a new force of nature with unknown strength). The p-value, as well as the ratio of the likelihood under the null to the maximized likelihood under the alternative, can both strongly disfavor the null, while the Bayesian posterior probability for the null can be arbitrarily large. The professional statistics literature has many impassioned comments on the paradox, yet there is no consensus either on its relevance to scientific communication or on the correct resolution. I believe that the paradox is quite relevant to frontier research in high energy physics, where the model assumptions can evidently be quite different from those in other sciences. This paper is an attempt to explain the situation to both physicists and statisticians, in hopes that further progress can be made.

This paradox isn’t a paradox at all; the different approaches give different answers because they ask different questions. Both could be right, but I firmly believe that one of them answers the wrong question.

Astronomy (and Particle Physics) Look-alikes, No. 92

Posted in Astronomy Lookalikes with tags , , on April 24, 2014 by telescoper

Although it’s not strictly an astronomical observation, I am struck by the resemblance between the distinguished particle physicist and blogger Professor Alfred E. Neuman, of University College London, and the iconic cover boy of Mad Magazine, Jon Butterworth. This could explain a lot about the Large Hadron Collider.

PP_Lookalike

Spoof Positive?

Posted in Science Politics, Uncategorized with tags , on June 29, 2010 by telescoper

Only time for a brief post today, as I’m shortly off to London for some external examining in the East End.

It’s interesting that yesterday’s #SpoofJenks day generated so many contributions that the Guardian decided to get the main instigator, particle physicist Jon Butterworth, to write about it on their Science Blog.  My own contribution of yesterday gets a mention there too.

I have to say I found the whole thing very amusing and wholeheartedly agree with Jon Butterworth (whose original spoof started it all off) when he explained that his primary aim was more to let off steam and less to try to persuade Simon Jenkins of the error of his ways. I felt the same way. Better to poke fun back than allow him to get to you.

I didn’t feel parody was necessary in Simon Jenkins’ case because his arguments are full of factual errors and non sequitur. In fact, it did occur to me that his piece might be deliberately ironic. Could one of the prime movers behind the Millennium Dome really be serious when he talks about the wastefulness of CERN? Perhaps he’s spoofed us all. But even that wouldn’t excuse his snide personal attack on Martin Rees.

Anyway, as you will have noticed, I  just went for straight mockery and had a good half-an-hour of lunchtime fun writing it.  A few people seem to have liked my piece, but at least one blogger found it “unpleasant”. You can’t win ’em all. For what it’s worth, I still think he deserved it.

In fact, I posted a considerably less vitriolic reaction to Jenkins article on Saturday, but trying to respond in rational terms was something I found very frustrating. Only a few hundred people read this blog so it’s pretty futile to try to take on a columnist from a national newspaper that’s read by hundreds of thousands. I’m not sure he’s listening anyway, as he’s written similar drivel countless times before. Far better, in my view, to join the collective piss-taking. At least it got the Guardian interested.

Maybe after all this Jenkins will actually engage in a dialogue with scientists instead of merely insulting them? Perhaps. But I’m not holding my breath.