Archive for Economics

Irreproducibility and Public Trust

Posted in Bad Statistics, Books with tags , , , , , , on April 8, 2026 by telescoper

A news item from Maynooth University led me to a paper published in Nature with the title Investigating the replicability of the social and behavioural sciences, co-authored by Dermot Lynott, a colleague from the Psychology Department. The abstract reads:

Pursuing replicability — independent evidence for previous claims — is important for creating generalizable knowledge1,2. Here we attempted replications of 274 claims of positive results from 164 quantitative papers published from 2009 to 2018 in 54 journals in the social and behavioural sciences. Replications were high powered on average to detect the original effect size (median of 99.6%), used original materials when relevant and available, and were peer reviewed in advance through a standardized internal protocol. Replications showed statistically significant results in the original pattern for 151 of 274 claims (55.1% (95% confidence interval (CI) 49.2–60.9%)) and for 80.8 of 164 papers (49.3% (95% CI 43.8–54.7%)), weighed for replicating multiple claims per paper. We observed modest variation in replication rates across disciplines (42.5–63.1%), although some estimates had high uncertainty. The median Pearson’s r effect size was 0.25 (95% CI 0.21–0.27) for original studies and 0.10 (95% CI 0.09–0.13) for replication studies, an 82.4% (95% CI 67.8–88.2%) reduction in shared variance. Thirteen methods for evaluating replication success provided estimates ranging from 28.6% to 74.8% (median of 49.3%). Some decline in effect size and significance is expected based on power to detect original effects and regression to the mean because we replicated only positive results. We observe that challenges for replicability extend across social–behavioural sciences, illustrating the importance of identifying conditions that promote or inhibit replicability3,4.

The outcome of this study is that only about half of claimed positive results in such areas as psychology, economics and education were found to be reproducible. The only thing that surpises me about the result is that it is as high as 50%. And before you get snarky about “soft sciences”, there is a similar phenomenon in physics and astronomy too.  Physicist John Bahcall famously said that, based on his experience, “about half of all 3σ detections are false”.

I think at least some of the irreproducible results stem from inappropriate statistical reasoning and/or the incorrect interpretation of statistical evidence. I’ve published a number of examples of such things on this blog (e.g. here and here). I also wrote a book some years ago trying to explain the centrality of statistical reasoning to science generally (though from a perspective based on my background in cosmology). I thought I would rehash and publish here some paragraphs from the end of that book that touch on public trust in science. I was fairly optimistic then, but things are undoubtedly far worse now. We’re seeing widespread cuts to research funding in the United States, the UK and many other countries.

–0–

Anyway, I thought I’d take the opportunity to re-iterate why I statistics and statistical reasoning are so important to science. In fact, I think they lie at the very core of the scientific method, although I am still surprised how few practising scientists are comfortable with even basic statistical language. A more important problem is the popular impression that science is about facts and absolute truths. It isn’t. It’s a process. In order to advance it has to question itself. Getting this message wrong – whether by error or on purpose -is immensely dangerous.

Statistical reasoning also applies to many facets of everyday life, including business, commerce, transport, the media, and politics. Probability even plays a role in personal relationships, though mostly at a subconscious level. It is a feature of everyday life that science and technology are deeply embedded in every aspect of what we do each day. Science has given us greater levels of comfort, better health care, and a plethora of labour-saving devices. It has also given us unprecedented ability to destroy the environment and each other, whether through accident or design.

Civilized societies face rigorous challenges in this century. We must confront the threat of climate change and forthcoming energy crises. We must find better ways of resolving conflicts peacefully lest nuclear or conventional weapons lead us to global catastrophe. We must stop large-scale pollution or systematic destruction of the biosphere that nurtures us. And we must do all of these things without abandoning the many positive things that science has brought us. Abandoning science and rationality by retreating into religious or political fundamentalism would be a catastrophe for humanity.

Unfortunately, recent decades have seen a wholesale breakdown of trust between scientists and the public at large. This is due partly to the deliberate abuse of science for immoral purposes, and partly to the sheer carelessness with which various agencies have exploited scientific discoveries without proper evaluation of the risks involved. The abuse of statistical arguments have undoubtedly contributed to the suspicion with which many individuals view science.

There is an increasing alienation between scientists and the general public. Many fewer students enrol for courses in physics and chemistry than a a few decades ago. Fewer graduates mean fewer qualified science teachers in schools. This is a vicious cycle that threatens our future. It must be broken.

The danger is that the decreasing level of understanding of science in society means that knowledge (as well as its consequent power) becomes concentrated in the minds of a few individuals. This could have dire consequences for the future of our democracy.Very few politicians are scientifically literate. How can we expect to control the application of science when the necessary understanding rests with an unelected “priesthood” that is hardly understood by, or represented in, our democratic institutions?

Very few journalists or television producers know enough about science to report sensibly on the latest discoveries or controversies. As a result, important matters that the public needs to know about do not appear at all in the media, or if they do it is in such a garbled fashion that they do more harm than good.

Years ago I used to listen to radio interviews with scientists on the Today programme on BBC Radio 4. I even did such an interview once. It is a deeply frustrating experience. The scientist usually starts by explaining what the discovery is about in the way a scientist should, with careful statements of what is assumed, how the data is interpreted, and what other possible interpretations might be and the likely sources of error. The interviewer then loses patience and asks for a yes or no answer. The scientist tries to continue, but is badgered. Either the interview ends as a row, or the scientist ends up stating a grossly oversimplified version of the story.

Some scientists offer the oversimplified version at the outset, of course, and these are the ones that contribute to the image of scientists as priests. Such individuals often believe in their theories in exactly the same way that some people believe religiously. Not with the conditional and possibly temporary belief that characterizes the scientific method, but with the unquestioning fervour of an unthinking zealot. This approach may pay off for the individual in the short term, in popular esteem and media recognition – but when it goes wrong it is science as a whole that suffers. When a result that has been proclaimed certain is later shown to be false, the result is widespread disillusionment.

The worst example of this tendency that I can think of is the constant use of the phrase “Mind of God” by theoretical physicists to describe fundamental theories. This is not only meaningless but also damaging. As scientists we should know better than to use it. Our theories do not represent absolute truths: they are just the best we can do with the available data and the limited powers of the human mind. We believe in our theories, but only to the extent that we need to accept working hypotheses in order to make progress. Our approach is pragmatic rather than idealistic. We should be humble and avoid making extravagant claims that can’t be justified either theoretically or experimentally.

The more that people get used to the image of “scientist as priest” the more dissatisfied they are with real science. Most of the questions asked of scientists simply can’t be answered with “yes” or “no”. This leaves many with the impression that science is very vague and subjective. The public also tend to lose faith in science when it is unable to come up with quick answers. Science is a process, a way of looking at problems not a list of ready-made answers to impossible problems. Of course it is sometimes vague, but I think it is vague in a rational way and that’s what makes it worthwhile. It is also the reason why science has led to so many objectively measurable advances in our understanding of the world.

I realise I must sound very gloomy about this, but I do think there are good prospects that the gap between science and society may gradually be healed. The fact that the public distrust scientists leads many of them to question us, which is a very good thing. They should question us and we should be prepared to answer them. If they ask us why, we should be prepared to give reasons. If enough scientists engage in this process then what will emerge is and understanding of the enduring value of science. I don’t just mean through the DVD players and computer games science has given us, but through its cultural impact. It is part of human nature to question our place in the Universe, so science is part of what we are. It gives us purpose. But it also shows us a way of living our lives. With some notabnle exceptions, the scientific community is open, internationally-minded, and imbued with a philosophy of cooperation. It values reason and looks to the future rather than the past. Like anyone else, scientists will always make mistakes, but we can always learn from them. The logic of science may not be infallible, but it’s probably the best logic there is in a world so filled with uncertainty.

4 Comments »

The Terror of Maths

Posted in History, The Universe and Stuff with tags , , , , on May 9, 2016 by telescoper

I’m not sure whether to be amused or appalled by the story of the Professor whose flight was delayed in order for him to be interrogated because a fellow passenger saw him doing some mathematical calculations. I know some people who find mathematics scary but that’s taking things too far! I wonder if the passenger was Simon Jenkins?

I was wondering whether the calculation was concerned with plane geometry but that seems not to be the case. The academic concerned is an Economist and he was studying a differential equation. That surprises me. I hadn’t realised economists knew about calculus. Or about anything else, for that matter.

The BBC coverage of the story used the following image:

scary_maths

The physicists among you will recognize this as a representation of some of Maxwell’s Equations. I very much doubt they played a part in the work of  our Economics Professor, so presumably this is just one of the  BBC’s stock of generic “scary maths” images.

Other things worth noting are that this version of Maxwell’s Equations isn’t written in SI units, the standard notation in the UK and Europe. As a matter of fact it uses cgs units, which suggests it may be an American import. Nor is it really correct anyway, because the time derivative inside the brackets should surely be partial.

All of which goes to demonstrate how Mathematics is usually viewed in the media and, by extension, the public at large: like an arcane book written in an incomprehensible  language that should be viewed with suspicion or ridicule by any sensible person.

There is nothing new about this, of course. I’m reminded that in 1870, during the Franco-Prussian Way, Norwegian mathematician Sophus Lie was arrested in France on suspicion of being a German spy because the authorities thought his mathematical notes were coded messages of some sort.

In reality, mathematics is the most open and universal language of all and, as such, is a powerful force for human good. Among many other things, quantitative reasoning and proper logic help to defend us against those who lie and distort the facts in order to gain power. Mathematics may not be the easiest language to learn, but it’s well worth the effort, even if you can only master the basics.

 

 

4 Comments »

Why is Astronomy Important?

Posted in The Universe and Stuff with tags , , , , on November 5, 2013 by telescoper

There’s an interesting and unusual article on the arXiv today entitled Why is Astronomy Important? Here is the abstract:

For a long time astronomers and other scientists believed that the importance of their work was evident to society. But in these difficult days of financial austerity, even the most obvious benefits of science have to undergo careful scrutiny. Eradicating poverty and hunger is a worldwide priority, and activities that do not directly attempt to resolve these issues can be hard to justify and support. However, several studies have told us that investing in science education, research and technology provides a great return not only economically, but culturally and indirectly for the population in general and has helped countries to face and overcome crises. The scientific and technological development of a country or region is closely linked to its human development index a statistic that is a measure of life expectancy, education and income.

The full text of the paper can be found on the IAU website here.

The article focusses on matters relating to the transfer of technology between astronomy and, e.g. industry, aerospace, and medicine, its effect on technology we are familiar with in everyday life, on astronomy as an exemplar of international collaboration and on its wider cultural and philosophical impact. Many of the points made in this article can also be found in the Royal Astronomical Society‘s free publication Beyond the Stars: Why Astronomy Matters which is available for free online here.

I recommend you read the full article and make your own mind up about why astronomy is important. I have just two comments, which are partly questions. The first is that I’ve always had a bit of a problem with the interpretation of correlations like that mentioned in the last sentence of the abstract (between technological development and the human development index). The issue is the basic one that correlation of two phenomena does not necessarily imply that one causes the other. Is it really possible to establish rigorously a causal link between spending money on astronomy and wider societal benefits? I’m not saying that there isn’t such a link, just that it’s difficult to interpret evidence which is dependent on so many factors. Could one not argue instead that more developed countries spend more money on astronomy because they can afford to?

The other thing that troubles me with arguments of the type presented in the paper is that there is a danger that  emphasizing the transfer of knowledge to other disciplines as the rationale for funding astronomy implicitly negates the argument that astronomy has intrinsic worth of its own. In other words, answering the question “Why is Astronomy important?” seems to accept at the outset that it isn’t.  If it is indeed the case that we can only justify astronomy because it has produced spin-offs in, e.g., medicine, why not just spend more money on medicine and forget the astronomy?

I’m not saying that the technology transfer arguments carry no weight, just that they are definitely double-edged and should be used with caution. For the record, I think we should fund Astronomy (and other sciences) primarily because they are an essential part of the fabric of our culture and civilization; all the rest is icing on the cake. In other words, I support state funding for the sciences for very much the same reasons as for the arts.  I’m fully aware, however, that this unlikely to persuade the powers that be as effectively as an appeal to economic benefits; that’s why science funding has fared so much better than arts funding in this age of austerity.

14 Comments »

Fears for the Future

Posted in Finance, Politics, Science Politics with tags , , on April 10, 2010 by telescoper

Just came back from a lovely cycle ride to find that my polling card arrived through my letterbox while I was out. Gordon Brown announced the election earlier this week, so it’s quite impressive how efficiently the electoral system swings into action.  It’s a pity so much else is screwed up.  Anyway, Parliament now goes into limbo and we have three weeks of heightened tedium to endure while the politicians try to convince us that, despite all the mess they’ve made of things so far, they do actually know what they’re doing.

I still don’t know how I’m going to cast my vote on May 6th (polling day). I can’t see myself voting for the incumbents – for more reasons than I have time to list. My experience of Thatcher’s Britain in the 1980s convinced me that I’ll never vote Conservative either. And the Lib Dems are just, well, a bit pathetic. I will vote. I just don’t know who I’ll vote for. I’ll have to look at my constituency’s history carefully to see if tactical voting might help. Perhaps more on that in due course…

Anyway, whatever the result of the election turns out to be, I’m pretty scared about what the next three or four years has in store.  The huge budget deficit that the government has built up saving the banks from collapse is going to have to be dealt with. The recent budget didn’t really do anything to tackle it, but everybody knows that was just a holding operation until the election is over. Whoever takes power afterwards will have to take serious measures to fix things. It won’t be pretty. Tax rises and public spending cuts are both inevitable as  the international bond markets threaten to downgrade Britains AAA credit rating. If that happens we will end up with runaway debts and increasingly expensive borrowing.  Don’t think we won’t go the way of Greece.

In the meantime our economy is carrying on as if it is in a trance. House prices continue to rise, the FTSE index is climbing, interest rates are at the astonishingly low level of 0.5%. It can’t possibly go on. Houses are clearly still overvalued, at immense social cost to people wanting to start a family. The stock market is gaining because investors are not getting any return from cash deposits, and companies are boosting their profits by sacking staff and cutting costs rather than generating new demand. As soon as interest rates go up again – which they surely must – I think there’s a good chance the stock market will fall again. If you don’t hold any shares yourself you may think that’s not important. However, it directly affects the pensions of millions of people, most of whom are not wealthy, because that’s where a lot of their pension schemes’ money is invested.

The most pressing issue is not who wins the election but whether there is a winner. If the election turns out indecisively – which at the moment seems quite likely – then we’re going to see turmoil on a scale that makes the banking nightmare of 2007 look like a tea party. And even if there is an outright winner, there’s no guarantee that they will have the gumption to even begin tackling the problem.

Of course, as a scientist working in a University, I’m also concerned about what’s going to happen to my own livelihood after the election. The recent mess this government has made of science funding has blotted its record on this, which was previously not bad. However, the true scale of this country’s economic problems seems to be too much for our political leaders, both present and future, to cope with. I don’t see any of the parties having the vision to manage the current crisis as well as putting together a coherent plan to build a better future. I’m not the only person to think so, in fact, as a letter in The Times today from a group of distinguished astronomers made clear. Other nations (especially the USA and France) are all investing heavily in science as a means to secure future economic growth. We’ve already started cutting back, and don’t see any strong political voice to reverse that policy.

Of course people don’t just vote for their immediate self-interest. Science is important to me, and I think it’s important for the country too, but there are other issues. There’s more to life than economics too. This country has been in a post-Imperial sleepwalk for too long and it needs to snap out of it. We need to renew our political system, which has grown distant and unaccountable. We need to deal with a looming energy crisis. We need to develop a proper education system that is fit for the 21st century. And we need to deal with the problems of a rapidly ageing population. For these reasons, and more, I hope the next Parliament will contain politicians with the vision necessary to see this country through the tough times ahead. Unfortunately, I don’t think it will.

I’m just glad I’m no longer young.

15 Comments »