February | 2009 | In the Dark

Archive for February, 2009

False Positives

Posted in Bad Statistics with tags Bayesian, cancer screening, medical statistics on February 27, 2009 by telescoper

There was an interesting article on the BBC website this week that, for once, contains an example of a reasonable discussion of statistics in the mass media. I’m indebted to my friend Anton for pointing it out to me. I’ve filed it along with examples of Bad Statistics because the issue is usually poorly explained. I don’t think the article itself is bad. In fact, it’s rather good.

The question is all about cancer screening, specifically for breast cancer, but the lesson could apply to a host of other situations. In the original context, the question goes as follows:

Say that routine screening is 90% accurate. Say you have a positive test. What’s the chance that your positive test is accurate and you really have cancer?

Presumably there will be many of you that think the answer is 90%. Hands up if you think this!

If you don’t think it’s 90% then what do you think it is?

The correct answer is that you have no idea. I haven’t given you enough information.

To see why, imagine that the prevalence of cancer in the population is such that 1% of a randomly selected sample will have it. Out of a thousand people one would expect that, on average, ten would have cancer. If the test is 90% accurate then 9 of these will show positive signs and only one won’t.

However, 990 people out of the original thousand don’t have cancer. If the test is only 90% accurate then 10%, i.e. 99 of these will show a false positive.

Thus the total number of positive tests is 108 and only 9 of the individuals concerned actually have cancer. The odds are therefore 9/108. That’s only about a 1-in-12 chance that you have cancer.

But that depends on my assumption about the overall rate in the population. If that number is different it changes the odds. Without this information, the problem is ill-posed.

The more general way of looking at this is in terms of conditional probabilities. What you are given is that P(positive test| cancer)=P(+|C)=0.9 and P(negative test|no cancer)=0.9, while P(negative test|cancer)= 0.1 and P(positive test|no cancer)=P(+|N)=0.1. What you want to know is P(cancer|positive test)=P(C|+). This can be obtained from Bayes’ Theorem but only if you know P(cancer)=P(C)=1-P(N), since people either have cancer or they don’t.

The answer is given by P(C|+)=P(C)P(+|C)/[P(C)P(+|C)+P(N)P(+|N)], which for the numbers I gave above= 0.01 x 0.9/[0.01 x 0.9 + 0.99 x 0.1]=0.009/[0.009+0.099], which gives the same answer as before.

So the moral is that you shouldn’t panic if you get a positive test from a screening test of this type. As long as the condition being tested is relatively rarer than the likelihood of an error in the test result then the chances are high that you’ve got nothing to worry about. But of course, you should take more detailed tests.

The Bayesian way is the easy way!

4 Comments »

A Welsh Affair

Posted in Science Politics with tags HEFCE, HEFCW, Institute of Welsh Affairs, Research Assessment Exercise, Welsh Assembly Government on February 26, 2009 by telescoper

Today I had the “pleasure” of attending a day-long conference called Funding, Risk and Innovation: Wales’s Engagement with Science Policy organized by the Institute of Welsh Affairs and sponsored by, amongst other organizations, the Institute of Physics. I had hoped that this would give me an insight into the landscape of Welsh science politics which might bear fruit in the future. As if.

Unfortunately, but alas predictably, there wasn’t much of interest. I think the first presentation of the day perfectly illustrated the whole problem. Opening the batting was Ieuan Wyn Jones, Deputy First Minister and Minister for Economic Development, from the Welsh Assembly Government or WAG. He gave a motherhood-and-apple-pie talk about how important science was to the future of Wales, took a few questions and then left. Those of us scientists who had gone to the meeting hoping for engagement between politicians and scientists were left to discuss things between ourselves. Hardly the point.

Next was Phil Gummett, Chief Executive of HEFCW who gave the results of the latest Research Assessment Exercise (which I’ve blogged about here, there and everywhere). To my dismay he announced that HEFCW are indeed going to use the 0:1:3:7 weighting formula adopted by HEFCE, but has found a bit more cash which it will add to the pot of money allocated to 4*. However, unlike in the case of English universities, HEFCW is not going to apply any protection to STEM subjects (Science, Technology & Medicine). In the case of my own department at Cardiff University, which got a very low assessment of 4* research, this is very bad news.

When I got home this evening I read the same news in the Times Higher. I could have found this out without wasting a day sitting in a ghastly conference room in the soulless Cardiff Novotel. Still, the lunch wasn’t bad.

Phil Gummett struck me as quite a reasonable chap who is trying to do the right thing, but whose hands are tied by the Welsh Assembly which has decided that Higher Education in Wales is not as high a priority as Further Education, with the result that the funds available to HEFCW for research is less than it would be for English universities. University STEM departments in Wales altogether receive about £10M less from HEFCW than they would get from HEFCE if they were in England. For physics, this will probably get worse after the 2008 RAE.

The reason for this pessimism is that, as I’ve noted before, Physics did rather badly in the RAE compared to other discplines, with a significantly lower fraction of work assessed at 4* (world-leading). Since the funding formula is heavily weighted by the 4* element, physics will suffer relative to other disciplines. HEFCW will not attempt to correct this. I think the Chair of the Physics panel, Sir John Pendry, must shoulder at least some of the blame for the gross anomaly that this represents.

It remains to be seen what happens to physics nationally, but I fear the RAE may undo a decade of very effective positive campaigning about the importance of physics. I’ve already heard from various Heads of Physics departments around the country (even those who have done well in the RAE) who have been asked by their Vice-Chancellors why they have done so much worse than other disciplines.

The final thing he said was that HEFCW would make its allocations as block grants to the Universities concerned and that they should make their own decisions as to how to allocate the funds to the departments. This sort of thing always annoys me. It’s admitting that the formula is probably stupid, so passing the buck to the institutions to sort out the mess themselves.

I spoke to a nice lady from Cardiff University’s planning department in the afternoon who said that they weren’t sure how they were going to allocate funds to Schools after the HEFCW grant was announced, and that the University as a whole was probably going to lose out in research funds, despite having 54% of all the 4* research in Wales.

The big problem is the funding gap caused by the WAG’s policy. Devolution has had a negative effect in science funding in Wales, while in Scotland it has had the opposite effect. The Scottish parliament seems much more interested in science than does the Welsh Assembly. Indeed, per capita, Scottish Universities have a much heavier level of research investment even than those in England, which in turn are much higher than in Wales.

EPSRC‘ recently allocated £82M to UK universities to fund doctoral training centres. In all, it allocated grants to 45 universities. Wales has 5% of the UK population, but not a single grant went to a Welsh university. Of the 1200 or so students these centres will train, not a single one will be in Wales. I can’t believe the Scottish assembly would have let such an outcome happen in Scotland.

Further strangulation of research funds is inevitable unless the WAG is persuaded to change its mind about the importance of science. But if the politicians don’t stay to listen to the arguments, how will this happen?

Over lunch I chatted to various physicists from Swansea University. Several of them had come to the meeting, but I was the only representative from Cardiff. There was a strong steer from the RAE panel for physics in terms of closer collaboration so we chatted a bit about possibilities for that. I think the consensus was that we’re probably going to be bounced down the road anyway so the best way forward would be to come up with a plan of our own instead of having someone else’s.

I promise not to mention the RAE again, until the final allocations are published in April!

5 Comments »

Blackbird has Spoken

Posted in Biographical, Poetry with tags Aubade, Columbo, Philip Larkin, Poetry on February 25, 2009 by telescoper

Over the last few days we’ve been having something approximating springtime here in Cardiff. It has been sunny and quite warm, my garden has started to come to life, and the crocuses have appeared in Bute Park. It’s also getting to the time when I won’t feel guilty for walking home in daylight. Soon I’ll even be able to walk home through Bute Park, which closes when it gets dark, currently at 5.15.

I hope this all continues into a pleasant spring and summer, without the heavy continuous rain we had last year. I’m not betting on it though.

However, the clement weather has given me one headache recently. With sunrise happening a bit earlier and the good weather giving the local wildlife something to shout about, the dawn chorus has been waking me up around 4am.

Or, actually, it’s not so much a chorus as a solo. A very loud blackbird has taken to sitting right next to my bedroom window and singing at the top of its voice.

I’m very fond of blackbirds. Once while I was in the garden in my old house in Beeston, a blackbird flew onto a fence post about a yard away from me and sat there looking at me as I stood with a spade in my hand. I looked back. We looked at each other for ages, the blackbird turning its head every now and again so as to peer at me with a different eye. I slowly raised my arm and extended a palm. To my absolute delight the bird hopped onto my open hand. It stayed there only a minute or so, probably until it realised my fingers weren’t actually big fat worms like it thought. For that moment, though, I felt a bit like a latter-day St Francis of Assisi.

Blackbirds have a very attractive song, but this one seems particularly loud and he certainly does go on a bit. For about a week now I’ve been unable to get back to sleep after being woken by this critter, and instead got up and had a cup of tea while he says what he has to say. Columbo finds his song quite interesting too, although the bird is always out of reach…

Years ago, I used to suffer very badly from insomnia so being awake at 4am is not an unfamiliar experience to me, although it’s much nicer to be woken by birdsong than to be unable to sleep in the first place. This all reminded me of a devastatingly brilliant poem called Aubade and written by Philip Larkin that was published in the Times Literary Supplement in 1977. This is one of the last poems written by Larkin, and is undoubtedly one of the greatest.

Written by a jazz-loving bachelor who drank too much, someone not unlike myself in some respects, I found it uncanningly accurate in its depiction of the bleak thoughts that tend to engulf you when you’re alone and awake in the silence before dawn. But I can assure you the mood is a whole lot lighter when you have a blackbird (and a cat) for company!

I work all day, and get half drunk at night.
Waking at four to soundless dark, I stare.
In time the curtain edges will grow light.
Till then I see what’s really always there:
Unresting death, a whole day nearer now,
Making all thought impossible but how
And where and when I shall myself die.
Arid interrogation: yet the dread
Of dying, and being dead,
Flashes afresh to hold and horrify.

The mind blanks at the glare. Not in remorse
– The good not used, the love not given, time
Torn off unused – nor wretchedly because
An only life can take so long to climb
Clear of its wrong beginnings, and may never:
But at the total emptiness forever,
The sure extinction that we travel to
And shall be lost in always. Not to be here,
Not to be anywhere,
And soon; nothing more terrible, nothing more true.

This is a special way of being afraid
No trick dispels. Religion used to try,
That vast moth-eaten musical brocade
Created to pretend we never die,
And specious stuff that says no rational being
Can fear a thing it cannot feel, not seeing
that this is what we fear – no sight, no sound,
No touch or taste or smell, nothing to think with,
Nothing to love or link with,
The anaesthetic from which none come round.

And so it stays just on the edge of vision,
A small unfocused blur, a standing chill
That slows each impulse down to indecision
Most things may never happen: this one will,
And realisation of it rages out
In furnace fear when we are caught without
People or drink. Courage is no good:
It means not scaring others. Being brave
Lets no-one off the grave.
Death is no different whined at than withstood.

Slowly light strengthens, and the room takes shape.
It stands plain as a wardrobe, what we know,
Have always known, know that we can’t escape
Yet can’t accept. One side will have to go.
Meanwhile telephones crouch, getting ready to ring
In locked-up offices, and all the uncaring
Intricate rented world begins to rouse.
The sky is white as clay, with no sun.
Work has to be done.
Postmen like doctors go from house to house.

Follow @telescoper

11 Comments »

Fat Tuesday

Posted in Jazz with tags Mardi Gras, New Orleans on February 24, 2009 by telescoper

Today’s the day we call in England Shrove Tuesday. We’re apparently all supposed to get shriven by doing a pennance before Lent . Another name for the occasion is Pancake Day, although I’m not sure what sort of pennance it is to be forced to eat pancakes.

Further afield the name for this day is a bit more glamorous. Mardi Gras, which I translated for the title using my schoolboy French, doesn’t make me think of pancakes but of carnivals. And being brought up in a house surrounded by Jazz, it makes me think of New Orleans and the wonderful marching bands that played not just during the Mardi Gras parades but at just about every occasion for which they could find an excuse, including funerals.

The Mardi Gras parades gave rise to many of the great tunes of New Orleans Jazz, many of them named after the streets through which the parade would travel, mainly in the famous French Quarter. Basin Street, South Rampart Street, and Bourbon Street are among the names redolent with history for Jazz fans and musicians around the world. I also remember a record by Humphrey Lyttelton‘s 1950s band called Fat Tuesday.

The New Orleans Mardi Gras has on recent occasions sometimes got a bit out of hand, and you probably wouldn’t want to take kids into the French Quarter for fear they would see things they shouldn’t. Personally, though, I’d love the chance to savour the atmosphere and watch the parades.

The clip I’ve chosen is of Bourbon Street Parade. The one and only time I went to New Orleans I felt a real thrill walking along this street, just because I’ve heard the tune so many times on old records. I didn’t go in Mardi Gras time, however, but in the middle of summer. The heat was sweltering and the humidity almost unbearable, but the air was filled with music as well as moisture. It was impossible to sleep in the heat, so I stayed up moving from bar to bar, drinking and listening to music until I was completely exhausted.

The tune was written by the late Paul Barbarin, who died in 1969 during a street parade in New Orleans. What a way to go. He also plays on the clip I included here.

I picked this particular clip because it features a much underrated British musician, Sammy Rimmington (although the notes on Youtube have muddled it up; he plays saxophone on this, not clarinet). My dad once played with Sammy Rimmington and I remember the unqualified admiration with which he (my dad) spoke of his (Sammy’s) playing.

10 Comments »

The Problem of the Steady State

Posted in Science Politics with tags astronomy, postgraduate, STFC on February 24, 2009 by telescoper

Just as a quick postscript to my recent item about proposed changes to the method of funding PhD students by STFC, let me point out the following simple calculation.

Assume that the number of permanent academic positions in a given field (e.g. astronomy) remains constant over time. If that is the case, each retirement (or other form of departure) from a permanent position will be replaced by one, presumably junior, scientist.

This means that over an academic career, on average, each academic will produce just one PhD who will get a permanent job. This of course doesn’t count students coming in from abroad, or those getting faculty positions abroad but in the case of the UK these are probably relatively small corrections.

Under the present supply of PhD studentships an academic can expect to get a PhD student at least once every three years or so. At a minimum, therefore, over a 30 year career one can expect to have ten PhD students. A great many supervisors have more PhD students than this, but this just makes the odds worse. The expectation is that only one of these will get a permanent job in the UK. The others (nine out of ten, according to my conservative estimate) above must either leave the field or the country to find permanent employment.

The arithmetic of this situation is a simple fact of life, but I’m not sure how many prospective PhD students are aware of it.

9 Comments »

Scientiae Doctores

Posted in Science Politics with tags astronomy, Bologna Agreement, postgraduate, STFC on February 22, 2009 by telescoper

The season for recruiting new research students is well and truly upon us and at the same the Science & Technology Facilities Council (STFC) is consulting about changing the way that it allocates PhD studentships to departments.

Most postgraduate students studying for PhDs in Astronomy are funded by STFC (although some Universities also fund their own internal studentships). The result of this arrangement is that successful applicants to a PhD course can receive a stipend which amounts to about £13K per annum. It’s not a huge amount of money, but it is a stipend rather than a salary so it’s tax-free. Since a PhD student also remains a student and therefore qualifies for various other fringe benefits (Council Tax, student discounts, etc), it’s not actually a bad deal for the student. Anyway, if it were significantly more then it’s possible PhD students would have to start paying back their student loans, which would make things worse. STFC also pays a tuition fee to the University concerned, but this is done directly and the student doesn’t even see that element of the funding.

Since about 1995, PPARC and then STFC has funded research studentships in areas within its remit by means of peer review. Departments have bid for studentships (every two years) and a panel awards an allocation depending on the quality of the bid. Of course, everyone asks for many more studentships than are available so what you get is a fraction of what you ask for. I wrote the application for the first ever quota studentships for the Astronomy group at the University of Nottingham, and did it again a couple of times after that. Each time, despite going into best bullshit mode to write the case, I was frustrated by the relatively small number of studentships we were awarded. Although we succeeded in building up gradually from zero to 2-3 per year, it was a very slow process.

In recent years, the funding mechanism has evolved slightly so that studentship fees and stipends were devolved to the departments concerned in terms of Doctoral Training Grants (DTGs) rather than being administered centrally by PPARC/STFC. In the old days, students used to get their stipend from PPARC/STFC whereas now they are paid by their department from a cash grant.

Anyway, for various reasons (chief among them being no doubt to save administrative costs) STFC has decided to consult on changes to the mechanism for allocating the DTGs to the various departments around the country. The most serious proposed change is to follow the practice at the Engineering and Physical Sciences Research Council (EPSRC) and dispense with peer review. Instead, the proposal is to award studentships based on a formula involving how successful the department is at obtaining postdoctoral research assistant (PDRA) support from STFC.

Here is the proposed formula:

Specifically, the studentship award per department should be proportional to the product of volume and average quality per academic within the department, that is to:

V * Q

The Committee has followed guidance in developing measures of V and Q that are non-subjective, repeatable and transparent. The volume V is defined as the number of academics (including Fellows) eligible to hold STFC research grants. The quality Q is measured by the number of STFC-funded PDRAs (P) awarded per academic (i.e. P/V), since this measures the success of the academic staff in securing STFC funding for PDRAs through peer-review. More precisely we define quality per academic as Q = [1 +(P/V)].

Although the Committee felt this definition of quality applied primarily to responsive-mode PDRAs, it agreed that PDRAs on project grants should be included, but with a weighting, relative to responsive-mode, of 0.33.

Using these definitions, the Committee recommends that the studentship award per department should be proportional to a simple product of volume and average quality per academic within the department, that is to:

N(students) µ V * Q

where Q = [1 + (P/V)]

And so the departmental quota is proportional to:

V[1+(P/V)] = V+P

In addition, recognising that very small departments offer more limited training opportunities on their own, a threshold is proposed, such that no studentships are awarded for V < 3. Instead, these very small departments/groups would be able to collaborate with other larger departments in seeking STFC studentship support.

Hence

N(students) µ V+P for V ³ 3

= 0 for V < 3

The constant of proportionality is chosen such that the total number of studentships equals the number available for allocation.

I think this is a fairly reasonable proposal, actually. The one thing I don’t really understand relates to the fact that STFC doesn’t just fund PDRAs on its grants, but under the Full Economic Cost regime (FEC), it also pays for fractions of academic staff effort for people working on its projects. On my recent successful STFC grant, for example, I was awarded 25% of my time (i.e. 0.25 FTE, full-time-equivalent) to do the research as well as a PDRA. Since the proposal above will have to cope with the question of what staff are “eligible” then why not make the quantity V proportional to the total FTEs funded, or at least only count those for whom some FEC time is allocated? And why not include staff FTE in the Q-factor too?

My guess is that such a modification wouldn’t make much difference to astronomy departments, but the original proposal has caused cries of anguish from particle physicists. This is because the number of PDRAs in particle physics is much smaller than in astronomy, so many large groups face a big reduction in their PhD quota. Including FEC numbers in the mix might well smooth the transition for them. For your information, the number of PDRAs per active astronomy researcher is around 0.5 at present.

Anyway, the deadline for consulting on this has passed (on February 20th) so we now wait to see what STFC actually does. Probably the consultation period is a purely cosmetic exercise anyway and what will emerge is exactly what was proposed.

If you ask me (and nobody did), all this is mere tinkering. I think there are serious problems with graduate funding in the UK and these require much more radical remedies. At the risk of (and indeed with the intention of) being provocative, here is my diagnosis and suggested remedies:

There are too many PhDs in astronomy. STFC funded 160 studentships in 2006, compared with 88 in 2000. There are nowhere near enough PDRA positions to accommodate this number of PhDs in academic research. And even those who get their first PDRA position have very limited prospects of getting a permanent job. The result is a generation of disaffected students employed as low-paid assistants for 3-4 years and then thrown aside when they have got their PhD.
Of course, applicants for PhD places don’t know what research is really like and some will leave academia of their own volition when they find out that it’s not for them. In my experience, though, most graduate applicants simply don’t realise how heavily the odds are stacked against them. Less than one in ten can possibly stay in research in the long term, and the more PhDs are funded the worse the odds against them become.
The short duration of a British PhD disadvantages our students with respect to those from the USA or continental europe, who all do a lengthy Masters course before taking their PhD. These take at least 5 years to complete. The result is that our home-grown PhDs are seriously disadvantaged in the job market against competitors from abroad. Similar points have been made forcefully by Ian Halliday.
My remedy is simple. Reduce the number of studentships but extend each one to five years and require each hosting department to provide a proper graduate school with intensive graduate-level courses to make up for the progressive reduction in content of undergraduate physics courses.
Even more unpopularly, I think the UK should scrap 4 years Masters (MPhys) programmes and embrace the structure of the Bologna agreement, i.e. a universal 3+2+3 structure of 3 years Bachelors, 2 years’ Masters and three years PhD.
Currently STFC stipends can only be paid to UK nationals and residents. It’s an open secret that most departments would preferentially recruit European physics graduates to their PhD positions if they were allowed to do so, because their undergraduate preparation is much better than that provided in UK universities. I propose that we abandon this protectionism and open up PhD opportunities to European applications, just as we would legally have to do if a PhD were considered to be a job.
Finally, I think the UK should consider the introduction of a common graduate entrance examination, perhaps based on the US GRE, to ensure the maintenance of appropriate standards for postgraduate entry and eligibility for STFC funding.

There are of course some advantages to the current British PhD system. For one thing, the PhD is earned very quickly. I was 25 when I got my PhD, and already had several publications. Most of my European collaborators were at least 30 before they got theirs (additional years have to be added for national service in many countries, but we don’t have it in the UK). But I am painfully aware that my technical knowledge outside the immediate area of my PhD is much thinner than most academics in the field. Now, in middle age, I feel like a long-distance runner who had inadequate preparation, went off too fast at the start of the race, and is now struggling along while people overtake him with monotonous regularity.

The nature of research in astronomy and cosmology has changed so much in the 20 years since I got my PhD that the old system has to go. Instead of tinkering with funding formula, driven principally by the need to save adminstrative costs within STFC, we need a radical overhaul of the entire graduate education system in the UK, involving all research councils and their political masters.

Unfortunately, though, for the time being at least the politicians have other more pressing matters to worry about, such the collapse of the economy.

9 Comments »

Anthropology

Posted in Jazz with tags Anthropology, bebop, Bud Powell on February 21, 2009 by telescoper

Just to balance the books here’s a wonderful version of the classic bebop tune Anthropology, performed by Bud Powell who I mentioned in my previous post about Thelonious Monk.

This really is authentic bebop, with its complex yet propulsive drum patterns and jagged melodic lines characterized by unusual intervals and little punctuating tags at the end of each phrase in the solo. Also brilliant, but quite different to Monk, Bud Powell was very much a direct translation onto the keyboard of the saxophone and trumpet styles of Charlie Parker and Dizzy Gillespie.

The tune was written by them too, although it uses essentially the same chords as George Gershwin’s I Got Rhythm. It was a standard bebop procedure to write a new melodic line on top of an existing harmonic structure and give the result a new name. Jazz improvisation is always based on the harmonies (chord changes) rather than the tune, so this is a way to build new compositions quickly with the knowledge that the harmonic foundations would be sound. In the 40s there was a classic Charlie Parker recording session during which the band decided to do such a variation of the standard Cherokee (written by Ray Noble). In the middle of the performance they absent-mindedly played the Cherokee theme and there was a cry of anguish from the control room by the Producer who obviously hoped that if they stayed off the actual tune of Cherokee he wouldn’t have to pay composer’s royalties. So that take was abandoned, and they did another. The final version was called Koko and it’s one of the Charlie Parker classics. Many other classic bop tunes were made in this way, with different standard tunes underneath them. Necessity and budget restrictions are the mothers of invention.

Oh, and just listen to the fantastic double bass by the brilliant (and then very young) Niels-Henning Orsted Pedersen who drives it along like the clappers!

When you read about the structure of bebop it seems so complicated and obscure that it’s hard to believe that the result is such a thrill to listen to. I think it is the highest point of twentieth century music-making, both in the creativity and skill of its proponents and in the sheer excitement of its sound which is made all the more remarkable when you realise how difficult it is to play!

2 Comments »

Eccentric Sphere

Posted in Jazz with tags Blue Monk, Thelonious Monk on February 21, 2009 by telescoper

Some weeks ago I posted a clip of Billie Holliday and Lester Young that was part of a 1958 TV programme called Sounds of Jazz. I found another clip from that show and decided to put it on here because it’s absolutely fascinating. The star this time is the great Thelonious Monk, whose middle name was Sphere.

Thelonious Monk was a unique musician. His remarkable self-taught style of piano playing was unlike that of anyone who came before him. Look at his hands in this clip and you’ll see that his fingers are straight as he plays: he uses them a bit like mallets in order to get such a percussive sound from the instrument (although the audio is a bit muffled on this track).

Monk was often called “The High Priest of Bop” and regarded as one of the leaders of the post-war bebop revolution in Jazz alongside Charlie Parker and Dizzy Gillespie. Actually, I don’t think Monk ever really played bebop: the archetypal bop pianist was undoubtedly Bud Powell whose style was quite different to Monk. But the “High Priest” tag owed at least something to his eccentric personality: he hardly ever spoke and, aside from his music, he communicated mainly through his choice of hat.

Monk’s piano style is hard to describe – his wife Nellie once described it as “Melodious Thunk” – but I’ve always loved his music. To me his solos sound like someone talking directly at you in a strange and wonderful language that you don’t quite understand but which sounds beautiful anyway. His use of syncopation is quite different from the usual bebop musicians and it seems, to me anyway, to echo the rhythms of everyday speech. But, above all, when you hear Monk play the piano, you know immediately who it is. He had many admirers, but nobody could play like him. He was a genius.

In later life his behaviour became disturbingly erratic; he would sometimes stand up in the middle of a performance and go wandering around the stage. His music also deteriorated, I think, from the early sixties onwards. His best records are from the 40s and 50s. I think it was generally assumed that he had a drugs problem, which he may well have had, but it was eventually realised that he was suffering from a serious mental illness. Although attempts were made to treat this, he stopped playing in the 1970s and lived out the rest of his as a recluse.

I remember very well the day he died in February 1982. It was during the Newcastle Jazz Festival and on the day when the great British jazz pianist Stan Tracey was due to give a concert there. As we took our seats in the Newcastle Playhouse for the gig, an announcement was made that Thelonious Monk had died. Stan Tracey, for whom Monk had been a major musical inspiration, responded to the occasion by playing two sets exclusively consisting of tunes by his hero. It was one of the best concerts I’ve ever been to and remains strong in my memory to this day.

That also reminds me to say that as well as being a completely unique and individualistic piano player, Thelonious Monk was also a great Jazz composer who penned some of the great modern standards of the idiom: ‘Round Midnight, Straight No Chaser, Epistrophy, the list is substantial. And then of course there is the tune he plays on this clip which became something of a signature tune.

Off the record, I don’t think it’s completely fair to say that Thelonious actually wrote Blue Monk. The main theme was in fact borrowed from an earlier tune called Pastel Blue written by the trumpeter Charlie Shavers. It’s basically a straightforward 12 bar blues, lacking the twists and turns and stops and starts of his more typical compositions, but Monk recorded more varied versions of this tune than practically any other. This clip is a short rendition, but it’s fascinating to see the other characters in shot. Sitting at Monk’s piano looking straight at him is none other than Count Basie, and we also get a glimpse of blues singer Joe Turner standing at the side. Later on, there’s a shot of the great saxophonist Coleman Hawkins clearly enjoying the music as I hope you will.

2 Comments »

Throwing a Fit

Posted in Bad Statistics, The Universe and Stuff with tags Cosmology, statistics on February 18, 2009 by telescoper

I’ve just been to a very interesting and stimulating seminar by Subir Sarkar from Oxford, who spoke about Cosmology Beyond the Standard Model, a talk into which he packed a huge number of provocative comments and interesting arguments. His abstract is here:

Precision observations of the cosmic microwave backround and of the large-scale clustering of galaxies have supposedly confirmed the indication from the Hubble diagram of Type Ia supernovae that the universe is dominated by some form of dark energy which is causing the expansion rate to accelerate. Although hailed as having established a ‘standard model’ for cosmology, this raises a profound problem for fundamental physics. I will discuss whether the observations can be equally well explained in alternative inhomogeneous cosmological models that do not require dark energy and will be tested by forthcoming observations.

He made no attempt to be balanced and objective, but it was a thoroughly enjoyable polemic making the point that it is possible that the dark energy whose presence we infer from cosmological observations might just be an artifact of using an oversimplified model to interpret the data. I actually agreed with quite a lot of what he said, and certainly think the subject needs people willing to question the somewhat shaky foundations on which the standard concordance cosmology is built.

But near the end, Subir almost spoiled the whole thing by making a comment that made me decide to make another entry in my Room 101 of statistical horrors. He was talking about the spectrum of fluctuations in the temperature of the Cosmic Microwave Background as measured by the Wilkinson Microwave Anisotropy Probe (WMAP):

I’ve mentioned the importance of this plot in previous posts. In his talk, Subir wanted to point out that the measured spectrum isn’t actually fit all that well by the concordance cosmology prediction shown by the solid line.

A simple way of measuring goodness-of-fit is to work out the value of chi-squared which relates to the sum of the squares of the residuals between the data and the fit. If you do this with the WMAP data you will find that the value of chi-squared is actually a bit high, so high indeed that there is only a 7 per cent chance of such a value arising in a concordance Universe. The reason is probably to do with the behaviour at low harmonics (i.e. large scales) where there are some points that do appear to lie off the model curve. This means that the best fit concordance model isn’t a really brilliant fit, but it is acceptable at the usual 5% significance level.

I won’t quibble with this number, although strictly speaking the data points aren’t entirely independent so the translation of chi-squared into a probability is not quite as easy as it may seem. I’d also stress that I think it is valuable to show that the concordance model isn’t by any means perfect. However, in Subir’s talk the chi-squared result morphed into a statement that the probability of the concordance model being right is only 7 per cent.

No! The probability of chi-squared given the model is 7%, but that’s quite different to the probability of the model given the value of chi-squared…

This is a thinly disguised example of the prosecutor’s fallacy which came up in my post about Sir Roy Meadow and his testimony in the case against Sally Clark that resulted in a wrongful conviction for the murder of her two children.

Of course the consequences of this polemicist’s fallacy aren’t so drastic. The Universe won’t go to prison. And it didn’t really spoil what was a fascinating talk. But it did confirm in my mind that statistics is like alcohol. It makes clever people say very silly things.

7 Comments »

Chinese Puzzles

Posted in Uncategorized with tags Azed, Dunhuang Star Chart, Royal Astronomical Society on February 17, 2009 by telescoper

On Friday 13th February I made one of my sporadic trips to London to go to the Royal Astronomical Society monthly meeting, catch up with friends, and dine at the Athenaeum with the RAS Club. That also gave me the excuse to stay in London over Valentine’s day and go with an old friend to the Opera followed by dinner in Chinatown on Saturday night.

As it happens, the RAS meeting also had a taste of the Orient about it because there were two absolutely fascinating talks about the Dunhuang Star Chart. This is a paper scroll found amongst many thousand similar objects squirrelled away by Buddhist monks in a tomb which was then subsequently bricked up and painted over. It lay undisturbed for a thousand years until rediscovered and basically plundered by treasure hunters, adventurers and archaeologists and its contents dispersed around the globe.

The Dunhuang Star Chart thus found its way to the British Library in London where it has recently been the subject of a special study involving both historians and astronomers. You can see this huge and very ancient sky map in full online here.

I had read about this sky map before in some book about the history of astronomy, but I hadn’t realised that its date had recently been re-evaluated to put it not in the 10th century (as I had previously believed), but in the middle of the 7th centur, possibly as early as 640 AD. Moreover, recent quite convincing mathematical analysis has shown that the chart is not just made of freehand sketches but was produced with some mathematical precision using a form of cylindrical projection. Once again, we find the Chinese were well in advance of their western counterparts in terms of scientific knowledge.

So why were these scrolls hidden away? There are two theories. One is that the monks were concerned about imminent invasion from the west and they simply wanted to safeguard their knowledge until it could be reclaimed. Unfortunately it never was. The other theory is based on the fact that astronomical knowledge was highly classified in this period of Chinese imperial rule, the Tang Dynasty. The astrological clues contained in star charts could be used to cast doubt on the Emperor if they fell into the wrong hands, so were forbidden to all but the inner court. Astronomy was Top Secret. The monks at Dunhuang may have hidden their papers because they shouldn’t have had them in the first place, and feared the wrath of the Emperor if they were discovered by the Imperial heavies.

I find mysterious artefacts like this absolutely fascinating and they also strengthen my conviction that astronomy and archaeology have much in common. Both are observational rather than experimental sciences, and both rely on making inferences based on indirect and sometimes scanty clues. Perhaps its this that makes both disciplines prone to a few flights of fancy every now and again as well as posing puzzles which perhaps will never be solved.

Anyway, topping the bill at the RAS was the President, Andy Fabian, whose Presidential Address was entitled Black Holes at Work. Unfortunately, the thing that didn’t work was the data projector so we had an embarassing delay while people rushed around trying to fix it. One of the charms of the RAS is that it never seems to be quite at the forefront of technology. Anyway, once he got going the talk was very interesting. He was short of time at the end, though, so I didn’t have time to ask the obligatory question about magnetic fields.

Then it was down to the Atheneaum and a nice dinner and rather a lot to drink.

The following evening after the Opera we went for dinner in Chinatown in Soho. The chilly West End streets were crowded, with what I originally assumed to be Valentine dates but which appeared instead to be mainly standard tourists taking advantage of the weak pound. Many restaurants were completely full, but eventually we found a table in a good place and all was well.

Coming back to Cardiff the following day I bought the Observer so I could do the crosswords on the train, and was reminded of the Azed competition crossword a couple of weeks ago which involved a quotation from a poem about St Valentine’s day by Coventry Padmore. It was quite a strange puzzle of a type called “Letters Latent” in which the cryptic part referred to the answer minus one or more letters. The quote concerned was

Well dost thou, Love, thy solemn Feast to hold
In vestal February;

the poem is trying to make the point that wintry February is a good time for St Valentine’s day as during spring and summer nobody needs to be reminded about the birds and the bees.

The task for competition entrants was to clue the word “vestal” in such a way that the definition referred to the whole word but the cryptic part omitted the s. My attempt was