This seems like a good day for reblogging, so try this for size. It gives instructions on when to believe stories about discoveries of exciting new physics by large consortia…
It’s an interesting piece, however it does seem to me that it gives necessary conditions for believing a result, but not sufficient ones. It’s not unknown for refereed articles to be wrong…
It’s been quite while since I posted a little piece about Bayesian probability. That one and the others that followed it (here and here) proved to be surprisingly popular so I’ve been planning to add a few more posts whenever I could find the time. Today I find myself in the office after spending the morning helping out with a very busy UCAS visit day, and it’s raining, so I thought I’d take the opportunity to write something before going home. I think I’ll do a short introduction to a topic I want to do a more technical treatment of in due course.
A particularly important feature of Bayesian reasoning is that it gives precise motivation to things that we are generally taught as rules of thumb. The most important of these is Ockham’s Razor. This famous principle of intellectual economy is variously presented in Latin as Pluralites non est ponenda sine necessitate or Entia non sunt multiplicanda praetor necessitatem. Either way, it means basically the same thing: the simplest theory which fits the data should be preferred.
William of Ockham, to whom this dictum is attributed, was an English Scholastic philosopher (probably) born at Ockham in Surrey in 1280. He joined the Franciscan order around 1300 and ended up studying theology in Oxford. He seems to have been an outspoken character, and was in fact summoned to Avignon in 1323 to account for his alleged heresies in front of the Pope, and was subsequently confined to a monastery from 1324 to 1328. He died in 1349.
In the framework of Bayesian inductive inference, it is possible to give precise reasons for adopting Ockham’s razor. To take a simple example, suppose we want to fit a curve to some data. In the presence of noise (or experimental error) which is inevitable, there is bound to be some sort of trade-off between goodness-of-fit and simplicity. If there is a lot of noise then a simple model is better: there is no point in trying to reproduce every bump and wiggle in the data with a new parameter or physical law because such features are likely to be features of the noise rather than the signal. On the other hand if there is very little noise, every feature in the data is real and your theory fails if it can’t explain it.
To go a bit further it is helpful to consider what happens when we generalize one theory by adding to it some extra parameters. Suppose we begin with a very simple theory, just involving one parameter 
Quantities such as
Obviously, having three degrees of freedom with which to describe the data should enable one to get a closer fit than is possible with just one. The greater flexibility within the general theory can be exploited to match the measurements more closely than the original. In other words, such a model can improve the likelihood, i.e. the probability of the obtained data arising (given the noise statistics – presumed known) if the signal is described by whatever model we have in mind.
But Bayes’ theorem tells us that there is a price to be paid for this flexibility, in that each new parameter has to have a prior probability assigned to it. This probability will generally be smeared out over a range of values where the experimental results (contained in the likelihood) subsequently show that the parameters don’t lie. Even if the extra parameters allow a better fit to the data, this dilution of the prior probability may result in the posterior probability being lower for the generalized theory than the simple one. The more parameters are involved, the bigger the space of prior possibilities for their values, and the harder it is for the improved likelihood to win out. Arbitrarily complicated theories are simply improbable. The best theory is the most probable one, i.e. the one for which the product of likelihood and prior is largest.
To give a more quantitative illustration of this consider a given model 
Now assume we have some data 
The interesting thing to look at now is not the best-fitting choice of model parameters 
This is just the normalizing constant 
In statistical mechanics things like
The usefulness of the Bayesian evidence emerges when we ask the question whether our
The answer is that although adding an extra degree of freedom can increase the first term in the integral defining 
This is a satisfying result that is in nice accord with common sense. But I think it goes much further than that. Many modern-day physicists are obsessed with the idea of a “Theory of Everything” (or TOE). Such a theory would entail the unification of all physical theories – all laws of Nature, if you like – into a single principle. An equally accurate description would then be available, in a single formula, of phenomena that are currently described by distinct theories with separate sets of parameters. Instead of textbooks on mechanics, quantum theory, gravity, electromagnetism, and so on, physics students would need just one book.
The physicist Stephen Hawking has described the quest for a TOE as like trying to read the Mind of God. I think that is silly. If a TOE is every constructed it will be the most economical available description of the Universe. Not the Mind of God. Just the best way we have of saving paper.
If you think the grass is greener on the EPSRC side of the fence than on the STFC one, then you should read this post by the genial e-Astronomer. Times are tough…
via The e-Astronomer
I’ve felt a mini-rant brewing for a few days now, as I’ve been reading through some of the interim reports my project students have written. I usually quite enjoy reading these, in fact. They’re not too long and I’m usually pretty impressed with how the students have set about the sometimes tricky things I’ve asked them to do. One pair, for example, is reanalysing the measurements made at the 1919 Eclipse expedition that I blogged about here, which is not only interesting from a historical point of view but which also poses an interesting challenge for budding data analysts.
So it’s not the fact that I have to read these things that annoys me, but the strange way students write them, i.e. almost entirely in the passive voice, e.g. “The experiment was calibrated using a phlogiston normalisation widget…”.
I accept that people disagree about whether the passive voice is good style or not. Some journals actively encourage the passive voice while others go the opposite way entirely . I’m not completely opposed to it, in fact, but I think it’s only useful either when the recipient of the action described in the sentence is more important than the agent, or when the agent is unknown or irrelevant. There’s nothing wrong with “My car has been stolen” (passive voice) since you would not be expected to know who stole it. On the other hand “My Hamster has been eaten by Freddy Starr” would not make a very good headline.
The point is that the construction of a statement in the passive voice in English is essentially periphrastic in that it almost inevitably involves some form of circumlocution – either using more words than necessary to express the meaning or being deliberately evasive by introducing ambiguity. Both of these failings should be avoided in scientific writing.
Apparently our laboratory instructors tell students to write their reports in the passive voice as a matter of course. I think this is just wrong. In a laboratory report the student should describe what he or she did. Saying what “was done” often leaves the statement open to the interpretation that somebody else did it. The whole point of a laboratory report is surely for the students to describe their own actions. “We calibrated the experiment..” is definitely to be preferred to the form I gave above.
Sometimes it is appropriate to use the passive voice because it is the correct grammatical construction in the circumstances. Sometimes also the text just seems to work better that way too. But having to read an entire document written in the passive voice drives me to distraction. It’s clumsy and dull.
In scientific papers, things are a little bit different but I still think using the active voice makes them easier to read and less likely to be ambiguous. In the introduction to a journal paper it’s quite acceptable to discuss the background to your work in the passive voice, e.g. “it is now generally accepted that…” but when describing what you and your co-authors have done it’s much better to use the active voice. “We observed ABC1234 using the Unfeasibly Large Telescope..” is, to my mind, much better than “Observations of ABC1234 were made using..”.
Reading back over this post I notice that I have jumped fairly freely between active and passive voice, thus demonstrating that I don’t have a dogmatic objection to its use. What I’m arguing is that it shouldn’t be the default, that’s all.
My guess is that a majority of experimental scientists won’t agree with this opinion, but a majority of astronomers and theoreticians will.
This guess will now be tested using a poll…
An opinion piece in New Scientist by Dan Hind (who apparently has a new book out) just caught my eye, and I couldn’t resist a quick comment.
The piece makes some good points. One is that much current science funding is actually nothing more than a direct subsidy for private industry. This troubles me. As I’ve blogged before, I think research that really is near-market shouldn’t be funded by the tax payer but by private investment by banks or venture capitalists. Publically funded science should be for speculative research that, regardless of whether it pays off commercially, is good from a cultural and intellectual perspective. I know I’m in a minority of my colleagues on this, but that’s what I think.
Where I disagree strongly with Dan Hind is the suggestion that science funding should be given
… to new bodies set up to allocate resources on the basis of a democratic vote. Scientists could apply to these bodies for funding and we could all have a say in what research is given support.
I can see that there are good intentions behind this suggestion, but in practice I think it would be a disaster. The problem is that the fraction of the general population that knows enough about science to make informed decisions about where to spend funding is just too small. That goes for the political establishment too.
If we left science funding to a democratic vote we’d be wasting vaulable taxpayer’s money on astrology, homeopathy and who knows what other kind of new age quackery. It’s true that the so-called experts get it wrong sometimes, but if left to the general public things would only get worse. I wish things were different, but this idea just wouldn’t work.
On the other hand, I don’t at all disagree with the motivation behind this suggestion. In an increasingly technologically-driven society, the gap between the few in and the many out of the know poses a grave threat to our existence as an open and inclusive democracy. The public needs to be better informed about science (as well as a great many other things). Two areas need attention.
In fields such as my own, astronomy, there’s a widespread culture of working very hard at outreach. This overarching term includes trying to get people interested in science and encouraging more kids to take it seriously at school and college, but also engaging directly with members of the public and institutions that represent them. Not all scientists take the same attitude, though, and we must try harder. Moves are being made to give more recognition to public engagement, but a drastic improvement is necessary if our aim is to make our society genuinely democratic.
But the biggest issue we have to confront is education. The quality of science education must improve, especially in state schools where pupils sometimes don’t have appropriately qualified teachers and so are unable to learn, e.g. physics, properly. The less wealthy are becoming systematically disenfranchised through their lack of access to the education they need to understand the complex issues relating to life in an advanced technological society.
If we improve school education, we may well get more graduates in STEM areas too although this government’s cuts to Higher Education make that unlikely. More science graduates would be good for many reasons, but I don’t think the greatest problem facing the UK is the lack of qualified scientists – it’s that too few ordinary citizens have even a vague understanding of what science is and how it works. They are therefore unable to participate in an informed way in discussions of some of the most important issues facing us in the 21st century.
We can’t expect everyone to be a science expert, but we do need higher levels of basic scientific literacy throughout our society. Unless this happens we will be increasingly vulnerable to manipulation by the dark forces of global capitalism via the media they control. You can see it happening already.
Having just given a lecture on probability and statistics to our first-year postgraduate students I thought I’d indulge in a bit of reflective practice (as the jargon goes) and make a few quick comments on teaching to see if I can generate some reaction. Part of the reason for doing this is that while I was munching my coffee and drinking my toast this morning – I’m never very coordinated first thing – I noticed an interesting post by a student on a blog that somehow wound up referring some traffic to one of my old posts about lecture notes.
I won’t repeat the entire content of my earlier discussion, but one of the main points I made was about how inefficient many students are at taking notes during lectures, so much so that the effort of copying things onto paper must surely prevent them absorbing the intellectual content of the lecture.
I dealt with this problem when I was an undergraduate by learning to write very quickly without looking at the paper as I did so. That way I didn’t waste time moving my head to and fro between paper and screen or blackboard. Of course, the notes I produced using this method weren’t exactly aesthetically pleasing, but my handwriting is awful at the best of times so that didn’t make much difference to me. I always wrote my notes up more neatly after the lecture anyway. But the great advantage was that I could write down everything in real time without this interfering with my ability to listen to what the lecturer was saying.
An alternative to this approach is to learn shorthand, or invent your own form of abbreviated language. This approach is, however, unlikely to help you take down mathematical equations quickly…
My experience nowadays is that students aren’t used to taking notes like this, so they struggle to cope with the old-fashioned chalk-and-talk style of teaching that some lecturers still prefer. That’s probably because they get much less practice at school than my generation. Most of my school education was done via the blackboard..
Nowadays, most lecturers use more “modern” methods than this. Many lecture using powerpoint, and often they give copies of the slides to students. Others give out complete sets of printed notes before, during, or after lectures. That’s all very well, I think, but what are the students supposed to be doing during the lecture if you do that? Listen, of course, but if there is to be a long-term benefit they should take notes too.
Even if I hand out copies of slides or other notes, I always encourage my students to make their own independent set of notes, as complete as possible. I don’t mean copying down what they see on the screen and what they may have on paper already, but trying to write down what I say as I say it. I don’t think many take that advice, which means much of the spoken illustrations and explanations I give don’t find their way into any long term record of the lecture.
And if the lecturer just reads out the printed notes, adding nothing by way of illustration or explanation, then the audience is bound to get bored very quickly.
My argument, then, is that regardless of what technology the lecturer uses, whether he/she gives out printed notes or not, then if the students can’t take notes accurately and efficiently then lecturing is a complete waste of time.
I like lecturing, because I like talking about physics and astronomy, but as I’ve got older I’ve become less convinced that lectures play a useful role in actually teaching anything. I think we should use lectures more sparingly, relying more on problem-based learning to instil proper understanding. When we do give lectures, they should focus much more on stimulating interest by being entertaining and thought-provoking. They should not be for the routine transmission of information, which is far too often the default.
Next year we’ll rolling out a new set of courses here in the School of Physics & Astronomy at Cardiff University. The express intent of this is to pare down the amount of material lectured to create more space for other types of activity, especially more exercise classes for problem-based learning. The only way to really learn physics is by doing it.
I’m not saying we should scrap lectures altogether. At the very least they have the advantage of giving the students a shared experience, which is good for networking and building a group identity. Some students probably get a lot out of lectures anyway, perhaps more than I did when I was their age. But different people benefit from different styles of teaching, so we need to move away from lecturing as the default option.
I don’t think I ever learned very much about physics from lectures, but I’m nevertheless glad I learned out how to take notes the way I did because I find it useful in all kinds of situations. Note-taking is a transferable skill, but it’s also a dying art.
Looking at my stats I find that my recent introductory post about Bayesian probability has proved surprisingly popular with readers, so I thought I’d follow it up with a brief discussion of some of the philosophical issues surrounding it.
It is ironic that the pioneers of probability theory, principally Laplace, unquestionably adopted a Bayesian rather than frequentist interpretation for his probabilities. Frequentism arose during the nineteenth century and held sway until recently. I recall giving a conference talk about Bayesian reasoning only to be heckled by the audience with comments about “new-fangled, trendy Bayesian methods”. Nothing could have been less apt. Probability theory pre-dates the rise of sampling theory and all the frequentist-inspired techniques that modern-day statisticians like to employ.
Most disturbing of all is the influence that frequentist and other non-Bayesian views of probability have had upon the development of a philosophy of science, which I believe has a strong element of inverse reasoning or inductivism in it. The argument about whether there is a role for this type of thought in science goes back at least as far as Roger Bacon who lived in the 13th Century. Much later the brilliant Scottish empiricist philosopher and enlightenment figure David Hume argued strongly against induction. Most modern anti-inductivists can be traced back to this source. Pierre Duhem has argued that theory and experiment never meet face-to-face because in reality there are hosts of auxiliary assumptions involved in making this comparison. This is nowadays called the Quine-Duhem thesis.
Actually, for a Bayesian this doesn’t pose a logical difficulty at all. All one has to do is set up prior probability distributions for the required parameters, calculate their posterior probabilities and then integrate over those that aren’t related to measurements. This is just an expanded version of the idea of marginalization, explained here.
Rudolf Carnap, a logical positivist, attempted to construct a complete theory of inductive reasoning which bears some relationship to Bayesian thought, but he failed to apply Bayes’ theorem in the correct way. Carnap distinguished between two types or probabilities – logical and factual. Bayesians don’t – and I don’t – think this is necessary. The Bayesian definition seems to me to be quite coherent on its own.
Other philosophers of science reject the notion that inductive reasoning has any epistemological value at all. This anti-inductivist stance, often somewhat misleadingly called deductivist (irrationalist would be a better description) is evident in the thinking of three of the most influential philosophers of science of the last century: Karl Popper, Thomas Kuhn and, most recently, Paul Feyerabend. Regardless of the ferocity of their arguments with each other, these have in common that at the core of their systems of thought likes the rejection of all forms of inductive reasoning. The line of thought that ended in this intellectual cul-de-sac began, as I stated above, with the work of the Scottish empiricist philosopher David Hume. For a thorough analysis of the anti-inductivists mentioned above and their obvious debt to Hume, see David Stove’s book Popper and After: Four Modern Irrationalists. I will just make a few inflammatory remarks here.
Karl Popper really began the modern era of science philosophy with his Logik der Forschung, which was published in 1934. There isn’t really much about (Bayesian) probability theory in this book, which is strange for a work which claims to be about the logic of science. Popper also managed to, on the one hand, accept probability theory (in its frequentist form), but on the other, to reject induction. I find it therefore very hard to make sense of his work at all. It is also clear that, at least outside Britain, Popper is not really taken seriously by many people as a philosopher. Inside Britain it is very different and I’m not at all sure I understand why. Nevertheless, in my experience, most working physicists seem to subscribe to some version of Popper’s basic philosophy.
Among the things Popper has claimed is that all observations are “theory-laden” and that “sense-data, untheoretical items of observation, simply do not exist”. I don’t think it is possible to defend this view, unless one asserts that numbers do not exist. Data are numbers. They can be incorporated in the form of propositions about parameters in any theoretical framework we like. It is of course true that the possibility space is theory-laden. It is a space of theories, after all. Theory does suggest what kinds of experiment should be done and what data is likely to be useful. But data can be used to update probabilities of anything.
Popper has also insisted that science is deductive rather than inductive. Part of this claim is just a semantic confusion. It is necessary at some point to deduce what the measurable consequences of a theory might be before one does any experiments, but that doesn’t mean the whole process of science is deductive. He does, however, reject the basic application of inductive reasoning in updating probabilities in the light of measured data; he asserts that no theory ever becomes more probable when evidence is found in its favour. Every scientific theory begins infinitely improbable, and is doomed to remain so.
Now there is a grain of truth in this, or can be if the space of possibilities is infinite. Standard methods for assigning priors often spread the unit total probability over an infinite space, leading to a prior probability which is formally zero. This is the problem of improper priors. But this is not a killer blow to Bayesianism. Even if the prior is not strictly normalizable, the posterior probability can be. In any case, given sufficient relevant data the cycle of experiment-measurement-update of probability assignment usually soon leaves the prior far behind. Data usually count in the end.
The idea by which Popper is best known is the dogma of falsification. According to this doctrine, a hypothesis is only said to be scientific if it is capable of being proved false. In real science certain “falsehood” and certain “truth” are almost never achieved. Theories are simply more probable or less probable than the alternatives on the market. The idea that experimental scientists struggle through their entire life simply to prove theorists wrong is a very strange one, although I definitely know some experimentalists who chase theories like lions chase gazelles. To a Bayesian, the right criterion is not falsifiability but testability, the ability of the theory to be rendered more or less probable using further data. Nevertheless, scientific theories generally do have untestable components. Any theory has its interpretation, which is the untestable baggage that we need to supply to make it comprehensible to us. But whatever can be tested can be scientific.
Popper’s work on the philosophical ideas that ultimately led to falsificationism began in Vienna, but the approach subsequently gained enormous popularity in western Europe. The American Thomas Kuhn later took up the anti-inductivist baton in his book The Structure of Scientific Revolutions. Kuhn is undoubtedly a first-rate historian of science and this book contains many perceptive analyses of episodes in the development of physics. His view of scientific progress is cyclic. It begins with a mass of confused observations and controversial theories, moves into a quiescent phase when one theory has triumphed over the others, and lapses into chaos again when the further testing exposes anomalies in the favoured theory. Kuhn adopted the word paradigm to describe the model that rules during the middle stage,
The history of science is littered with examples of this process, which is why so many scientists find Kuhn’s account in good accord with their experience. But there is a problem when attempts are made to fuse this historical observation into a philosophy based on anti-inductivism. Kuhn claims that we “have to relinquish the notion that changes of paradigm carry scientists ..closer and closer to the truth.” Einstein’s theory of relativity provides a closer fit to a wider range of observations than Newtonian mechanics, but in Kuhn’s view this success counts for nothing.
Paul Feyerabend has extended this anti-inductivist streak to its logical (though irrational) extreme. His approach has been dubbed “epistemological anarchism”, and it is clear that he believed that all theories are equally wrong. He is on record as stating that normal science is a fairytale, and that equal time and resources should be spent on “astrology, acupuncture and witchcraft”. He also categorised science alongside “religion, prostitution, and so on”. His thesis is basically that science is just one of many possible internally consistent views of the world, and that the choice between which of these views to adopt can only be made on socio-political grounds.
Feyerabend’s views could only have flourished in a society deeply disillusioned with science. Of course, many bad things have been done in science’s name, and many social institutions are deeply flawed. One can’t expect anything operated by people to run perfectly. It’s also quite reasonable to argue on ethical grounds which bits of science should be funded and which should not. But the bottom line is that science does have a firm methodological basis which distinguishes it from pseudo-science, the occult and new age silliness. Science is distinguished from other belief-systems by its rigorous application of inductive reasoning and its willingness to subject itself to experimental test. Not all science is done properly, of course, and bad science is as bad as anything.
The Bayesian interpretation of probability leads to a philosophy of science which is essentially epistemological rather than ontological. Probabilities are not “out there” in external reality, but in our minds, representing our imperfect knowledge and understanding. Scientific theories are not absolute truths. Our knowledge of reality is never certain, but we are able to reason consistently about which of our theories provides the best available description of what is known at any given time. If that description fails when more data are gathered, we move on, introducing new elements or abandoning the theory for an alternative. This process could go on forever. There may never be a final theory. But although the game might have no end, at least we know the rules….
Regular readers of this blog – both of them – will know that I engage, from time to time, in acitivities we are usually obliged to call “outreach”. For once, the OED gives a meaning which is pretty much spot on what is involved:
The activity of an organization in making contact and fostering relations with people unconnected with it, esp. for the purpose of support or education and for increasing awareness of the organization’s aims or message.
For us scientists this word means going out into the wider community (often, but not exclusively, in schools) and talking about our work to generate interest in science and also to explain what we do to the people who actually pay for it, i.e. taxpayers. This general activity is increasingly important for science generally, and astronomy has a particularly important part to play as it is not only very photogenic – there are lots of wonderful images to excite and engage the audience – but it also gives people the chance to think, at least for a while, about matters far removed from the mundanities of everyday life.
Most astronomy departments in Britain have very active outreach programmes and their importance is even recognized in, e.g., research grant applications, which nowadays require a statement about “public engagement” to be submitted alongside the technical science case.
I think colleagues in more mainstream subjects (e.g. condensed matter physics) often get a bit jealous of the astronomers in their department gallivanting off to all kinds of venues in the evenings to give talks. We usually say that what we’re doing is raising the profile of the discipline so that more students apply to do physics, justify our existence to the taxpayer, and so on. All very laudible things to do, but I have to admit I don’t really think of those things when I go off to do a popular talk. It’s not for the money either; I’m not famous enough to command a fee for public speaking, but even if I were I would probably waive it. The real reason I do these things because I like talking, especially about astrophysics, and especially to people who are interested enough to make a positive decision to come and listen.
Anyway, the reason for that rambling preamble about outreach was that yesterday I went to London to give a talk – in the upstairs room of a nice pub in the Marylebone area of Central London – for a group called Just Friends. When I was invited to give the talk I was a bit nervous because previous speakers had included David Starkey and Sir Roy Strong, as well as fellow astronomer and rising media star Marek Kukula, who actually turned up last night to lend me moral (?) support. My nervousness was, of course, entirely unjustified and the audience turned out to be very friendly.
I’ll leave it to the reader to work out what sort of people make up Just Friends, but there’s a bit of a clue in the list of previous speakers…
Because of the unusual location I had to take my own data projector along, which meant lugging it around all evening. Actually it wasn’t really “my own”. I borrowed it from a colleague in the department, so I was a bit nervous about breaking it as well as unsure how well it would work. As it happens, it all worked fine and I handed it back safely this morning.
Anyway, the room turned out to be very full and my talk – a variation on the theme of the Cosmic Web – seemed to go down quite well with an audience which was considerably less rowdy than I’d expected given that the talk was in a pub. After a fairly lengthy Q+A at the end and a quick drink, I headed back to Paddington for the 22.45 train back to Cardiff, getting home about 1.30am, exhausted but having enjoyed the evening enormously.
I suppose the price for having a captive audience and the chance to meet interesting new people at events like this is that you end up spreading yourself a bit thin. The morning after it feels less like outreach and more like overreach. The younger folk in the department seem to cope much better with all this scooting about. I’m definitely getting old.
I don’t have time to post much today so I thought I’d just put up a quick item about something that the e-astronomer (aka Andy Lawrence) has already blogged about, and generated a considerable amount of discussion about so I’ll just chip in with my two-penny-worth.
Some time ago I posted an item explaining how, in the run-up to last week’s Comprehensive Spending Review, the Royal Academy of Engineering had argued, in a letter to the Department of Business, Innovation and Skills (BIS), that government research funding should be
… concentrated on activities from which a contribution to the economy, within the short to medium term, is foreseeable. I recognise that this calls for significant changes in practice but I see no alternative in the next decade. This may mean disinvesting in some areas in order properly to invest in others.
They went on to say that
BIS should also consider the productivity of investment by discipline and then sub-discipline. Once the cost of facilities is taken into account it is evident that ‘Physics and Maths’ receive several times more expenditure per research active academic compared to those in ‘Engineering and Technology’. This ratio becomes significantly more extreme if the comparison is made between particle physics researchers and those in engineering and technology. Much of particle physics work is carried out at CERN and other overseas facilities and therefore makes a lower contribution to the intellectual infrastructure of the UK compared to other disciplines. Additionally, although particle physics research is important it makes only a modest contribution to the most important challenges facing society today, as compared with engineering and technology where almost all the research is directly or indirectly relevant to wealth creation.
I had hoped that this unseemly attack on particle physics would have been seen for what it was and would have faded into the background, but a recent article by Colin Macilwain has brought it back into the spotlight. I quote
UK engineers have started a scrap that will grow uglier as the spending cuts begin.
I should add that MacIlwain isn’t particularly supportive of the engineers’ position, but he does make some interesting remarks on the comparitively low status held by engineers in the United Kingdom compared to other countries, a point alsotaken up on Andy Lawrence’s blog. In my opinion this bare-faced attempt to feather their own nest at the expense of fundamental physics isn’t likely to generate many new admirers. Neither is the fact – and this is a point I’ve tried to make before – that the engineers’ argument simply doesn’t hold any water in the first place.
The point they are trying to make is that research in engineering is more likely to lead to rapid commercial exploitation than research in particle physics. That may be true, but it’s not a good argument for the government to increase the amount of research funding. If engineering and applied science really is “near market” in the way that the RAEng asserts, then it shouldn’t need research grants, but should instead be supported by venture capital or direct investment from industry. The financial acumen likely to be available from such investors will be much for useful for the commercial exploitation of any inventions or discoveries than a government-run research council. To be fair, as MacIlwain’s article explains, a large fraction of engineering research (perhaps 75%) is funded by commerce and industry. Moreover some engineering research is also too speculative for the market to touch and therefore does merits state support. However, that part that needs state support needs it for precisely the same reason that particle physics does, i.e. that its potential is long-term rather than short term. This means that is in the same boat as fundamental physics and shouldn’t keep pretending that it isn’t. If engineering research needs government funding then ipso facto it’s not likely to generate profits in the short term.
I think scientists and engineers would all be better off if they worked together to emphasize the amazingly successful links between fundamental physics and technology, as demonstrated by, e.g., the Large Hadron Collider at CERN and the mutual interdependence of their disciplines.
United we stand, and all that…
It’s been a while since I posed anything in the Bad Statistics file so I thought I’d return to the subject of one of my very first blog posts, although I’ll take a different tack this time and introduce it with different, though related, example.
The topic is forensic statistics, which has been involved in some high-profile cases and which demonstrates how careful probabilistic reasoning is needed to understand scientific evidence. A good example is the use of DNA profiling evidence. Typically, this involves the comparison of two samples: one from an unknown source (evidence, such as blood or semen, collected at the scene of a crime) and a known or reference sample, such as a blood or saliva sample from a suspect. If the DNA profiles obtained from the two samples are indistinguishable then they are said to “match” and this evidence can be used in court as indicating that the suspect was in fact the origin of the sample.
In courtroom dramas, DNA matches are usually presented as being very definitive. In fact, the strength of the evidence varies very widely depending on the circumstances. If the DNA profile of the suspect or evidence consists of a combination of traits that is very rare in the population at large then the evidence can be very strong that the suspect was the contributor. If the DNA profile is not so rare then it becomes more likely that both samples match simply by chance. This probabilistic aspect makes it very important to understand the logic of the argument very carefully.
So how does it all work? A DNA profile is not a complete map of the entire genetic code contained within the cells of an individual, which would be such an enormous amount of information that it would be impractical to use it in court. Instead, a profile consists of a few (perhaps half-a-dozen) pieces of this information called alleles. An allele is one of the possible codings of DNA of the same gene at a given position (or locus) on one of the chromosomes in a cell. A single gene may, for example, determine the colour of the blossom produced by a flower; more often genes act in concert with other genes to determine the physical properties of an organism. The overall physical appearance of an individual organism, i.e. any of its particular traits, is called the phenotype and it is controlled, at least to some extent, by the set of alleles that the individual possesses. In the simplest cases, however, a single gene controls a given attribute. The gene that controls the colour of a flower will have different versions: one might produce blue flowers, another red, and so on. These different versions of a given gene are called alleles.
Some organisms contain two copies of each gene; these are said to be diploid. These copies can either be both the same, in which case the organism is homozygous, or different in which case it is heterozygous; in the latter case it possesses two different alleles for the same gene. Phenotypes for a given allele may be either dominant or recessive (although not all are characterized in this way). For example, suppose the dominated and recessive alleles are called A and a, respectively. If a phenotype is dominant then the presence of one associated allele in the pair is sufficient for the associated trait to be displayed, i.e. AA, aA and Aa will both show the same phenotype. If it is recessive, both alleles must be of the type associated with that phenotype so only aa will lead to the corresponding traits being visible.
Now we get to the probabilistic aspect of this. Suppose we want to know what the frequency of an allele is in the population, which translates into the probability that it is selected when a random individual is extracted. The argument that is needed is essentially statistical. During reproduction, the offspring assemble their alleles from those of their parents. Suppose that the alleles for any given individual are chosen independently. If p is the frequency of the dominant gene and q is the frequency of the recessive one, then we can immediately write:
Using the product law for probabilities, and assuming independence, the probability of homozygous dominant pairing (i.e. AA) is p2, while that of the pairing aa is q2. The probability of the heterozygotic outcome is 2pq (the two possibilities, each of probability pq are Aa and aA). This leads to the result that
This called the Hardy-Weinberg law. It can easily be extended to cases where there are two or more alleles, but I won’t go through the details here.
Now what we have to do is examine the DNA of a particular individual and see how it compares with what is known about the population. Suppose we take one locus to start with, and the individual turns out to be homozygotic: the two alleles at that locus are the same. In the population at large the frequency of that allele might be, say, 0.6. The probability that this combination arises “by chance” is therefore 0.6 times 0.6, or 0.36. Now move to the next locus, where the individual profile has two different alleles. The frequency of one is 0.25 and that of the other is 0.75. so the probability of the combination is “2pq”, which is 0.375. The probability of a match at both these loci is therefore 0.36 times 0.375, or 13.5%. The addition of further loci gradually refines the profile, so the corresponding probability reduces.
This is a perfectly bona fide statistical argument, provided the assumptions made about population genetic are correct. Let us suppose that a profile of 7 loci – a typical number for the kind of profiling used in the courts – leads to a probability of one in ten thousand of a match for a “randomly selected” individual. Now suppose the profile of our suspect matches that of the sample left at the crime scene. This means that, either the suspect left the trace there, or an unlikely coincidence happened: that, by a 1:10,000 chance, our suspect just happened to match the evidence.
This kind of result is often quoted in the newspapers as meaning that there is only a 1 in 10,000 chance that someone other than the suspect contributed the sample or, in other words, that the odds against the suspect being innocent are ten thousand to one against. Such statements are gross misrepresentations of the logic, but they have become so commonplace that they have acquired their own name: the Prosecutor’s Fallacy.
To see why this is a fallacy, i.e. why it is wrong, imagine that whatever crime we are talking about took place in a big city with 1,000,000 inhabitants. How many people in this city would have DNA that matches the profile? Answer: about 1 in 10,000 of them ,which comes to 100. Our suspect is one. In the absence of any other information, the odds are therefore roughly 100:1 against him being guilty rather than 10,000:1 in favour. In realistic cases there will of course be additional evidence that excludes the other 99 potential suspects, so it is incorrect to claim that a DNA match actually provides evidence of innocence. This converse argument has been dubbed the Defence Fallacy, but nevertheless it shows that statements about probability need to be phrased very carefully if they are to be understood properly.
All this brings me to the tragedy that I blogged about in 2008. In 1999, Mrs Sally Clark was tried and convicted for the murder of her two sons Christopher, who died aged 10 weeks in 1996, and Harry who was only eight weeks old when he died in 1998. Sudden infant deaths are sadly not as uncommon as one might have hoped: about one in eight thousand families experience such a nightmare. But what was unusual in this case was that after the second death in Mrs Clark’s family, the distinguished paediatrician Sir Roy Meadows was asked by the police to investigate the circumstances surrounding both her losses. Based on his report, Sally Clark was put on trial for murder. Sir Roy was called as an expert witness. Largely because of his testimony, Mrs Clark was convicted and sentenced to prison.
After much campaigning, she was released by the Court of Appeal in 2003. She was innocent all along. On top of the loss of her sons, the courts had deprived her of her liberty for four years. Sally Clark died in 2007 from alcohol poisoning, after having apparently taken to the bottle after three years of wrongful imprisonment.The whole episode was a tragedy and a disgrace to the legal profession.
I am not going to imply that Sir Roy Meadows bears sole responsibility for this fiasco, because there were many difficulties in Mrs Clark’s trial. One of the main issues raised on Appeal was that the pathologist working with the prosecution had failed to disclose evidence that Harry was suffering from an infection at the time he died. Nevertheless, what Professor Meadows said on oath was so shockingly stupid that he fully deserves the vilification with which he was greeted after the trial. Two other women had also been imprisoned in similar circumstances, as a result of his intervention.
At the core of the prosecution’s case was a probabilistic argument that would have been torn to shreds had any competent statistician been called to the witness box. Sadly, the defence counsel seemed to believe it as much as the jury did, and it was never rebutted. Sir Roy stated, correctly, that the odds of a baby dying of sudden infant death syndrome (or “cot death”) in an affluent, non-smoking family like Sally Clarks, were about 8,543 to one against. He then presented the probability of this happening twice in a family as being this number squared, or 73 million to one against. In the minds of the jury this became the odds against Mrs Clark being innocent of a crime.
That this argument was not effectively challenged at the trial is truly staggering.
Remember that the product rule for combining probabilities
only reduces to
if the two events A and B are independent, i.e. that the occurrence of one event has no effect on the probability of the other. Nobody knows for sure what causes cot deaths, but there is every reason to believe that there might be inherited or environmental factors that might cause such deaths to be more frequent in some families than in others. In other words, sudden infant deaths might be correlated rather than independent. Furthermore, there is data about the frequency of multiple infant deaths in families. The conditional frequency of a second such event following an earlier one is not one in eight thousand or so, it’s just one in 77. This is hard evidence that should have been presented to the jury. It wasn’t.
Note that this testimony counts as doubly-bad statistics. It not only deploys the Prosecutor’s Fallacy, but applies it to what was an incorrect calculation in the first place!
Defending himself, Professor Meadows tried to explain that he hadn’t really understood the statistical argument he was presenting, but was merely repeating for the benefit of the court something he had read, which turned out to have been in a report that had not been even published at the time of the trial. He said
To me it was like I was quoting from a radiologist’s report or a piece of pathology. I was quoting the statistics, I wasn’t pretending to be a statistician.
I always thought that expert witnesses were suppose to testify about those things that they were experts about, rather than subjecting the jury second-hand flummery. Perhaps expert witnesses enjoy their status so much that they feel they can’t make mistakes about anything.
Subsequent to Mrs Clark’s release, Sir Roy Meadows was summoned to appear in front of a disciplinary tribunal at the General Medical Council. At the end of the hearing he was found guilty of serious professional misconduct, and struck off the medical register. Since he is retired anyway, this seems to me to be scant punishment. The judges and barristers who should have been alert to this miscarriage of justice have escaped censure altogether.
Although I am pleased that Professor Meadows has been disciplined in this fashion, I also hope that the General Medical Council does not think that hanging one individual out to dry will solve this problem. I addition, I think the politicians and legal system should look very hard at what went wrong in this case (and others of its type) to see how the probabilistic arguments that are essential in the days of forensic science can be properly incorporated in a rational system of justice. At the moment there is no agreed protocol for evaluating scientific evidence before it is presented to court. It is likely that such a body might have prevented the case of Mrs Clark from ever coming to trial. Scientists frequently seek the opinions of lawyers when they need to, but lawyers seem happy to handle scientific arguments themselves even when they don’t understand them at all.
I end with a quote from a press release produced by the Royal Statistical Society in the aftermath of this case:
Although many scientists have some familiarity with statistical methods, statistics remains a specialised area. The Society urges the Courts to ensure that statistical evidence is presented only by appropriately qualified statistical experts, as would be the case for any other form of expert evidence.
As far as I know, the criminal justice system has yet to implement such safeguards.
In the Dark 