Archive for the Science Politics Category

Newer Entries »

Cerebral Asymmetry: is it all in the Mind?

Posted in Bad Statistics, Science Politics with tags , , on November 12, 2008 by telescoper

After blogging a few days ago about the possibility that our entire Universe might be asymmetric, I found out today that a short comment of mine about a completely different form of asymmetry has been published in the Proceedings of the National Academy of Sciences of New York.

Earlier this summer a paper by Ivanka Savic & Per Lindstrom concerning gender and sexuality differences in brain structure received widespread press coverage and the odd blog comment. They had analysed a group of 90 volunteers divided into four classes based on gender and sexual orientation: male heterosexual, male homosexual, female heterosexual and female homosexual.

They studied the brain structure of these volunteers using Magnetic Resonance Imaging and used their data to look for differences between the different classes. In particular they measured the asymmetry between left and right hemispheres for their samples. The right side of the brain for heterosexual men was found to be typically about 2% larger than the left; homosexual women also had an asymmetry, but slightly smaller than this at about 1%. Gay men and heterosexual women showed no discernible cerebral asymmetry. These claims are obviously very interesting and potentially important if they turn out to be true. It is in the nature of the scientific method that such results should be subjected to rigorous scrutiny in order to check their credibility.

As someone who knows nothing about neurobiology but one or two things about statistics, I dug out the research paper by Savic & Lindstrom and looked at the analysis it presents. I very quickly began to suspect there might be a problem. For each volunteer, the authors obtain measurements of the left and right cerebral volumes (call these L and R respectively). Each pair of measurements is then combined to form an asymmetry index (AI) as (L-R)/(L+R). There is then a set of values for AI, one for each volunteer. The claim is that these are systematically different for the different gender and orientation groups, based on a battery of tests including Analysis of Variance (ANOVA) and t-tests based on sample means.

Of course, it would be better to do this using a consistent, Bayesian, approach because this would make explicit the dependence of the results on an underlying model of the data. Sadly, the statistical methodology available off-the-shelf is of inferior frequentist type and this is what researchers tend to do when they don’t really know what they’re doing. They also don’t bother to read the health warnings that state the assumptions behind the results.

The problem in this case is that the tests done by Savic & Lindstrom all depend on the quantity being analysed (AI) having a normal (Gaussian) distribution. This is very often a reasonable hypothesis for biometric data, but unfortunately in this case the construction of the asymmetry index is such that it is expected to have a very non-Gaussian shape as is commonly the case for distributions of variables formed as ratios. In fact, the ratio of two normal variates has a peculiar distribution with very long tails. Many statistical analyses appeal to the Central Limit Theorem to justify the assumption of normality, but distributions with very long tails (such as the Cauchy distribution) violate the conditions of this Theorem, namely that the distribution must have finite variance. The asymmetry index is probably therefore an inappropriate choice of variable for the tests that Savic & Lindstrom perform. In particular the significance levels (or p-values) quoted in their paper are very low (of order 0.0008, for example, in the ANOVA test) which is surprising for such small samples. These probabilities are obtained by assuming the observations have Gaussian statistics, and they would be much lower for a distribution with longer tails.

Being a friendly chap I emailed Dr Savic drawing this problem to her attention and asking if she knew about this problem and the possible implications it might have for the analysis she had presented. If not, I offered to do an independent (private) check on the data to see how reliable the claimed statistical results actually were. I never received a reply.

Worried that the world might be jumping to all kinds of far-reaching conclusions about gay genes based on these questionable statistics, I wrote instead to the editor of the Journal Proceedings of the National Academy of Sciences of New York, Randy Schekman, who suggested I submit a written comment to the Journal. I did, it was accepted by the editorial committee, and it came out in the 11th November Issue. What I didn’t realise was that Savic & Lindstrom had actually prepared a reply and that this was published alongside my comment. I find it strange that I wasn’t told about this before publication but that aside, it is in principle quite reasonable to let the authors respond to criticisms like mine. Their response reveals that they completely missed the point of the danger of long-tailed distributions I mentioned above. They state that “when the sample size n is big the sampling distribution of the mean becomes approximately normal regardless of the distribution of the original variable“. Not if the distribution of the original variable has such a long tail it doesn’t! In fact, if the observations have a Cauchy distribution then so does the sampling distribution of the mean, whatever the size of sample. You can find this caveat spelled out in many places, including here. Savic & Lindstrom seem oblivous to this pitfall, even after I specifically pointed it out to them.

They also claim that a group size of n=30 is sufficient to be confident that the central limit theorem holds. A pity, then, that none of their groups is of that size. The overall sample is 90, but it is broken down into two groups of 20 and two of 25.

cerebral-asymmetry

(c) 2008 Academy of Sciences of New York

They also say that the measured AI distribution is actually normal anyway and give a plot (above). This shows all the AI values binned into one histogram. Since they don’t give any quantitative measures of goodness of fit, it’s hard to tell whether this has a normal distribution or not. One can, however, easily identify a group of five or six individuals that seem to form a separate group with larger AI values (the small peak to the right of the large peak). Since they don’t give histograms broken down by group it is impossible to be sure, but I would hazard a guess that these few individuals might be responsible for the entire result; remember that the entire sample has n only of 90.

More alarmingly, Savic & Lindstrom state in their reply that “one outlier” is omitted from this graph. Really? On what basis was the outlier rejected? The existence of outliers could be evidence of exactly the sort of problem I am worried about! Unless there was a known mistake in the measurement, this outlier should never have been omitted. They claim that the “recalculation of the data excluding this outlier does not change the results”. It find it difficult to believe that the removal of an outlier from such a small sample could not change the p-values!

In my note I made a few constructive suggestions as to how the difficulty might be circumvented, by Savic & Bergstrom have not followed any of them. Instead they report (without details of the p-values) having done some alternative, non-parametric, tests. These are all very well, but they don’t add very much if their p-values also assume Gaussian statistics. A better way to do this sort of thing robustly would be using Monte Carlo simulations.

The bottom line is that after this exchange of comments we haven’t really got anywhere and I still don’t know if the result is significant. I don’t really think it’s useful to go backwards and forwards through the journal, so I’ve emailed Dr Savic again asking for access to the numbers so I can check the statistics privately. In astronomy it is quite normal for people to make their data sets publically available, but that doesn’t seem to be the case in neurobiology. I’m not hopeful that they will reply, especially since they branded my comments “harsh” and “inappropriate”. Scientists should know how to take constructive criticism.

Their conclusion may eventually turn out to be right, but the analysis done so far is certainly not robust and it needs further checking. In the meantime I don’t just have doubts about the claimed significance of this specific result, which merely serves to illustrate the extremely poor level of statistical understanding displayed by large numbers of professional researchers. This was one of the things I wrote about in my book From Cosmos to Chaos. I’m very confident that a large fraction of claimed results in biosciences are based on bogus analyses.

I’ve long thought that scientific journals that deal with subjects like this should employ panels of statisticians to do the analysis independently of the authors and also that publication of the paper should require publication of the raw data. Science advances when results are subject to open criticism and independent analysis. I sincerely hope that Savic & Lindstrom will release their data in order for their conclusions to be checked in this way.

It’s no wonder that there is so much public distrust of science, when such important claims are rushed into the public domain without proper scrutiny.

15 Comments »

The New Inflationary Universe

Posted in Finance, Science Politics on October 14, 2008 by telescoper

Among the bits of economic information released by the Office of National Statistics today is one item that academics in all disciplines wanted to hear about: the value of the Retail Prices Index (RPI) in the UK for September 2008, which turned out to be 5.0%.

The reason for the fascination with this number is that, in an unusual spasm of farsightedness, the University and College Union stipulated that the final stage of the pay deal it negotiated in 2006 would be applied in October 2008 and this would amount to 2.5% or the RPI whichever is the greater. Two years ago it seemed a very different world, and 2.5 % seemed to be much the likelier eventuality, but energy and commodity prices surged last year and the RPI now stands at double that figure. So we’re all set for a 5% pay rise this month, although probably we won’t actually get any more money until the November pay packet arrives.

It would have been even better had UCU chosen the Consumer Prices Index (CPI), which has now overtaken the RPI and stands at 5.2%. This is the governments preferred measure of inflation, which is based only on the price of consumer goods and household utilities, while the RPI includes other items such as mortgage costs and transport costs.

At least in the short-term, this seems good news for all academics in UK universities.

But even in paradise there was a serpent, and there is a significant danger that some departments’ balance sheets will suffer very badly from these extra salary costs. Many already operate on very tight margins. In the longer term there may be mergers and closures followed by redundancies. Also since the research councils’ cash allocations for the next few years are already fixed, an increase in salaries over that already accounted for will mean a corresponding reduction in the number of positions that can be funded, which is bad news for younger people looking for PDRA positions. Given that the Science and Technology Facilities Council‘s budget wasn’t very generous in the first place, causing a crisis in funding for astronomy and particle physics research the extra wage demands are likely to cause further strain.

Still, a 5% pay rise just before Xmas will be good while it lasts.

Leave a comment »

Wakeham Review

Posted in Science Politics, The Universe and Stuff with tags , , on October 1, 2008 by telescoper

Today is the day of publication of the Wakeham Review of the state of Physics in the United Kingdom. This report was commissioned by the Secretary of State for Innovation, Universities and Skills (DIUS) against the backdrop of the funding crisis that threatened to engulf the
Science and Technology Facilities Council (STFC) in December 2007 and which has led to drastic cuts in research grant funding in particle physics and astronomy throughout the country.

I started blogging a bit too late to join in the chorus of anger surrounding the handling of this crisis by STFC and especially by the behaviour of its Chief Executive, Keith Mason. An investigation of this by a parliamentary Select Committee stated that

“Substantial and urgent changes are now needed in the way in which the Council is run in order to restore confidence and to give it the leadership it desperately needs and has so far failed properly to receive”

If anyone was ever given a clear message that he should resign, this was it. But Keith Mason remains Chief Executive of STFC.

I hoped, therefore, to find some comment about this state of affairs in the Wakeham Review. I haven’t had time to read all of it, but most of it seems bland and rather self-congratulatory. It does, however, describe the strengths of astronomy and space science research in the UK, which is one of the areas placed in jeopardy by STFC’s cack-handed management and woeful lack of political nous. On the other hand, the UK has less impressive impact in other areas. Condensed matter physics was the research area in which most University-based physicists in the UK worked in 2001but their impact, at least in bibliometric terms, was and is unspectacular compared with other countries. Perhaps this is the reason why the number of condensed matter physicists submitted to the Research Assessment Exercise in 2008 has declined, while astronomy and astrophysics have increased.

The Wakeham review does not come to any clear conclusions on why some areas of physics are more popular than others, citing as possibilities laboratory costs and difficulties of attracting people into cross-disciplinary areas like biophysics or nanoscience. Since I’m not a member I don’t have to mince words like the panel did. I think some fields are popular because they are more interesting. And if people wanted to do chemistry or biology they wouldn’t have become physicists in the first place.

There are two paragraphs specifically about STFC, and they make very specific proposals although falling short of asking the current leadership to step down:

6. There is a need to ensure that there is coherence of planning of physics facilities and the allocation of physics research grants, so that research needs are closely aligned with facility provision. For that reason it is not desirable to separate former PPARC-like physics from the funding of its facilities. For this reason the Panel recommends that the current division of physics funding between Research Councils remains. Whilst recognising recent difficulties, the Panel believes that it is important that facilities provided for particle physics and astronomy researchers be directly tensioned with the budget for the research that will utilise those facilities. The current structure provides this tension in part of its remit. However, the panel believes that adding to this tension a further dimension of national facilities and a government Science and Innovation Campuses is just too much.

This is true but I think it’s only a small part of the problem.

The Panel recommends that:
a) the STFC be required at each CSR to bid for and allocate specific funds to former PPARC facilities and grant funding together.This would avoid the undesired tensioning of these grants and facilities support against national facilities and the project for the development of science and innovation campuses.

Good! But will this happen?

b) the existing structure should be allowed time to develop, given it was founded on the basis of extensive positive consultation. However, at an appropriate point following the review of STFC management currently being conducted by Dr David Grant, DIUS should commission a review to examine STFC operations.

*Sigh* Another review. Great.

The next one is a bit stronger:


7. The STFC’s governance structure must be representative of the community it serves in order to gain stakeholders’ confidence going forward.

“..stakeholders’ confidence going forward”? Ugh! Who wrote that bollocks?

The Panel believes that significant damage has been done to the UK’s international reputation in some areas of the discipline of physics following the furore that was generated by the manner, timescale of changes and announcement of recent STFC funding decisions.

You can say that again.

The Panel were very concerned at the make-up of the STFC Council, both in terms of the over representation of the executive and the lack of representation of the community it serves in comparison with other Research Councils. It is understood that this structure was deliberately adopted to deal with the distinct features of STFC that arose because of its multiple missions. However, this has not best served the scientific community in some branches of science whose input was at one level below the Council. This is in sharp distinction to the practice of other Research Councils.


The Panel recommends to DIUS that the membership of STFC’s Council be broadened to include more of the stakeholders in the science activity at the highest level, and to redress the balance between executive presence and non-executive oversight.

Somebody must have deleted the sentence about having to get a new Chief Executive.

I’m sure there’ll be a lot more on physics blogs when there’s been time to digest the whole report, and if I think I’ve missed anything at a first reading I may post some more myself.

1 Comment »

The MacGuffin Factor

Posted in Science Politics, The Universe and Stuff with tags , , , on September 22, 2008 by telescoper

Unpick the plot of any thriller or suspense movie and the chances are that somewhere within it you will find lurking at least one MacGuffin. This might be a tangible thing, such the eponymous sculpture of a Falcon in the archetypal noir classic The Maltese Falcon or it may be rather nebulous, like the “top secret plans” in Hitchcock’s The Thirty Nine Steps. Its true character may be never fully revealed, such as in the case of the glowing contents of the briefcase in Pulp Fiction , which is a classic example of the “undisclosed object” type of MacGuffin. Or it may be scarily obvious, like a doomsday machine or some other “Big Dumb Object” you might find in a science fiction thriller. It may even not be a real thing at all. It could be an event or an idea or even something that doesn’t exist in any real sense at all, such the fictitious decoy character George Kaplan in North by Northwest.

Whatever it is or is not, the MacGuffin is responsible for kick-starting the plot. It makes the characters embark upon the course of action they take as the tale begins to unfold. This plot device was particularly beloved by Alfred Hitchcock (who was responsible for introducing the word to the film industry). Hitchcock was however always at pains to ensure that the MacGuffin never played as an important a role in the mind of the audience as it did for the protagonists. As the plot twists and turns – as it usually does in such films – and its own momentum carries the story forward, the importance of the MacGuffin tends to fade, and by the end we have often forgotten all about it. Hitchcock’s movies rarely bother to explain their MacGuffin(s) in much detail and they often confuse the issue even further by mixing genuine MacGuffins with mere red herrings.

North by North West is a fine example of a multi-MacGuffin movie. The centre of its convoluted plot involves espionage and the smuggling of what is only cursorily described as “government secrets”. But although this is behind the whole story, it is the emerging romance, accidental betrayal and frantic rescue involving the lead characters played by Cary Grant and Eve Marie Saint that really engages the characters and the audience as the film gathers pace. The MacGuffin is a trigger, but it soon fades into the background as other factors take over.

There’s nothing particular new about the idea of a MacGuffin. I suppose the ultimate example is the Holy Grail in the tales of King Arthur and the Knights of the Round Table and, much more recently, the Da Vinci Code. The original Grail itself is basically a peg on which to hang a series of otherwise disconnected stories. It is barely mentioned once each individual story has started and, of course, is never found.

Physicists are fond of describing things as “The Holy Grail” of their subject, such as the Higgs Boson or gravitational waves. This always seemed to me to be an unfortunate description, as the Grail quest consumed a huge amount of resources in a predictably fruitless hunt for something whose significance could be seen to be dubious at the outset.The MacGuffin Effect nevertheless continues to reveal itself in science, although in different forms to those found in Hollywood.

The Large Hadron Collider (LHC), switched on to the accompaniment of great fanfares last week, provides a nice example of how the MacGuffin actually works pretty much backwards in the world of Big Science. To the public, the LHC was built to detect the Higgs Boson, a hypothetical beastie introduced to account for the masses of other particles. If it exists the high-energy collisions engineered by LHC should reveal its presence. The Higgs Boson is thus the LHC’s own MacGuffin. Or at least it would be if it were really the reason why LHC has been built. In fact there are dozens of experiments at CERN and many of them have very different motivations from the quest for the Higgs.

Particle physicists are not daft, however, and they have realised that the public and, perhaps more importantly government funding agencies, need to have a really big hook to hang such a big bag of money on. Hence the emergence of the Higgs as a sort of master MacGuffin, concocted specifically for public consumption, which is much more effective politically than the plethora of mini-MacGuffins which, to be honest, would be a fairer description of the real state of affairs.

Even this MacGuffin has its problems, though. The Higgs mechanism is notoriously difficult to explain to the public, so some have resorted to a less specific but more misleading version: “The Big Bang”. As I’ve already griped, the LHC will never generate energies anything like the Big Bang did, so I don’t have any time for the language of the “Big Bang Machine”, even as a MacGuffin.

While particle physicists might pretend to be doing cosmology, we astrophysicists have to contend with MacGuffins of our own. One of the most important discoveries we have made about the Universe in the last decade is that its expansion seems to be accelerating. Since gravity usually tugs on things and makes them slow down, the only explanation that we’ve thought of for this perverse situation is that there is something out there in empty space that pushes rather than pulls. This has various possible names, but Dark Energy is probably the most popular, adding an appropriately noirish edge to this particular MacGuffin. It has even taken over in prominence from its much older relative, Dark Matter, although that one is still very much around.

We have very little idea what Dark Energy is, where it comes from, or how it relates to other forms of energy we are more familiar with, so observational astronomers have jumped in with various grandiose strategies to find out more about it. This has spawned a booming industry in survey of the distant Universe (such as the Dark Energy Survey) all aimed ostensibly at unravelling the mystery of the Dark Energy. It seems that to get any funding at all for cosmology these days you have to sprinkle the phrase “Dark Energy” liberally throughout your grant applications.

The old-fashioned “observational” way of doing astronomy – by looking at things hard enough until something exciting appears (which it does with surprising regularity) – has been replaced by a more “experimental” approach, more like that of the LHC. We can no longer do deep surveys of galaxies to find out what’s out there. We have to do it “to constrain models of Dark Energy”. This is just one example of the not necessarily positive influence that particle physics has had on astronomy in recent times and it has been criticised very forcefully by Simon White.

Whatever the motivation for doing these projects now, they will undoubtedly lead to new discoveries. But my own view is that there will never be a solution of the Dark Energy problem until it is understood much better at a conceptual level, and that will probably mean major revisions of our theories of both gravity and matter. I venture to speculate that in twenty years or so people will look back on the obsession with Dark Energy with some amusement, as our theoretical language will have moved on sufficiently to make it seem irrelevant.

But that’s how it goes with MacGuffins. Even the Maltese Falcon turned out to be a fake in the end.

p.s. I heard on Saturday that the LHC is having some problems with its magnets and will actually be off-line for a few months. Last week I heard a particle physicist describing the great switch-on as like “Christmas”. This turns out to have been truer than he can have imagined. Only a week has passed and his most expensive toy is already broken…

8 Comments »

The Last Experiment

Posted in Science Politics, The Universe and Stuff with tags , , , on September 16, 2008 by telescoper

I’ve launched myself into the blogosphere just a bit too late for the feeding frenzy surrounding the switching on of the Large Hadron Collider at CERN last week. Obviously the event itself was a bit of a non-event as it will take years for anything interesting to come out the other end of its multi-billion-dollar tunnel. There are a couple of things worth saying in retrospect, though, now that the dust has settled.

The first is about all this nonsense concerning the creation of black holes that could destroy the Earth. If it were possible to create black holes in the LHC then they would be very puny ones, not capable of destroying anything very much at all. The phrase “black hole” conjures up Hollywood-style images of dead stars rampaging through the Galaxy devouring planets and costing a fortune in special effects. But not all black holes are massive enough to be stars in movies. If the LHC could make black holes it would only make very titchy ones. Since the gravitational effect of a black hole depends on its mass – and these little ones have very little of that – any that did pop out of an event in the LHC would be more of a pin-prick than a hole…

Moreover, energetic particles in the form of cosmic rays are constantly raining down on the Earth’s atmosphere, colliding with hadrons as they do so. The most extreme cosmic rays have energies far in excess of the limit that can be reached by the LHC. If an energetic hadron collision were going to produce a black hole that could destroy the planet, it would have happened a very long time ago and we wouldn’t be around to discuss the possibility.

So how did this daft idea come to dominate the news coverage surrounding the switch-on of the LHC? The press are never reluctant to peddle the doomsday scenario whenever they can as it appears to sell newspapers. But there is probably a bit more to it than that. I think part of it is a side-effect of the exaggerated language used by particle physicists in their attempt to use the LHC to capture the public imagination. “The Big Bang Machine” is just one example. If the experiment were really attempting to recreate the Big Bang, then there would indeed be much to be scared about. But the fact of the matter is that the LHC doesn’t reach energies anything like those reached in the Big Bang (nor even in the many smaller bangs that our Universe indulges in from time to time, such as supernova explosions).

The maximum energy reached by the LHC is going to be about 7 TeV (roughly equivalent to the energy of a bumble bee in flight). Although the very earliest stages of the Big Bang itself are not well understood, we are pretty sure that the primordial fireball started off with energies at least a million million (i.e. 1,000,000,000,000) times larger than this. It is doubtful (to say the least) that we’ll ever be able to build a device capable of reaching such energies, so the only “Big Bang Machine” there will ever be is the one we happen to be living in.

This is perhaps the reason why particle physicists are so desperate to glean maximum publicity for the LHC. It’s cost – though not extreme when compared to, for example, military spending – far exceeds that of any other scientific experiment. When it is over, will it be possible to build an even bigger experiment to probe even deeper into the subatomic world? Funding of such experiments generally comes from the public purse and it seems more than likely that the taxpayer will draw the line very soon. Although it won’t destroy the world, perhaps the LHC is nevertheless the end of the line for experimental physics of that kind.

So by all means let’s celebrate the LHC. It’s a wonderful demonstration of what international cooperation can achieve. It is also a response to the need all humans have to ask questions about our Universe. But let us not forget that our ability to probe the inner space of particles with experiments will always be limited, while the outer space beyond the stars offers much wider horizons.

PS. I can’t resist adding this link, as the best example of the worst of the hysteria about the LHC.

PPS. And this one, which explains why the LHC really is safe.

13 Comments »

Newer Entries »