The League of Small Samples
This morning I was just thinking that it’s been a while since I’ve filed anything in the category marked bad statistics when I glanced at today’s copy of the Times Higher and found something that’s given me an excuse to rectify my lapse. Today saw the publication of said organ’s new Student Experience Survey which ranks British Universities in order of the responses given by students to questions about various aspects of the teaching, social life and so on. Here are the main results, sorted in decreasing order:
| 1 | Loughborough University | 84.9 | 128 |
| 2 | University of Cambridge, The | 82.6 | 259 |
| 3 | University of Oxford, The | 82.6 | 197 |
| 4 | University of Sheffield, The | 82.3 | 196 |
| 5 | University of East Anglia, The | 82.1 | 122 |
| 6 | University of Wales, Aberystwyth | 82.1 | 97 |
| 7 | University of Leeds, The | 81.9 | 185 |
| 8 | University of Dundee, The | 80.8 | 75 |
| 9 | University of Southampton, The | 80.6 | 164 |
| 10 | University of Glasgow, The | 80.6 | 136 |
| 11 | University of Exeter, The | 80.3 | 160 |
| 12 | University of Durham | 80.3 | 189 |
| 13 | University of Leicester, The | 79.9 | 151 |
| 14 | University of St Andrews, The | 79.9 | 104 |
| 15 | University of Essex, The | 79.5 | 65 |
| 16 | University of Warwick, The | 79.5 | 190 |
| 17 | Cardiff University | 79.4 | 180 |
| 18 | University of Central Lancashire, The | 79.3 | 88 |
| 19 | University of Nottingham, The | 79.2 | 233 |
| 20 | University of Newcastle-upon-Tyne, The | 78.9 | 145 |
| 21 | University of Bath, The | 78.7 | 142 |
| 22 | University of Wales, Bangor | 78.7 | 43 |
| 23 | University of Edinburgh, The | 78.1 | 190 |
| 24 | University of Birmingham, The | 78.0 | 179 |
| 25 | University of Surrey, The | 77.8 | 100 |
| 26 | University of Sussex, The | 77.6 | 49 |
| 27 | University of Lancaster, The | 77.6 | 123 |
| 28 | University of Stirling, The | 77.6 | 44 |
| 29 | University of Wales, Swansea | 77.5 | 61 |
| 30 | University of Kent at Canterbury, The | 77.3 | 116 |
| 30 | University of Teesside, The | 77.3 | 127 |
| 32 | University of Hull, The | 77.2 | 87 |
| 33 | Robert Gordon University, The | 77.2 | 57 |
| 34 | University of Lincoln, The | 77.0 | 121 |
| 35 | Nottingham Trent University, The | 76.9 | 192 |
| 36 | University College Falmouth | 76.8 | 40 |
| 37 | University of Gloucestershire | 76.8 | 74 |
| 38 | University of Liverpool, The | 76.7 | 89 |
| 39 | University of Keele, The | 76.5 | 57 |
| 40 | University of Northumbria at Newcastle, The | 76.4 | 149 |
| 41 | University of Plymouth, The | 76.3 | 190 |
| 41 | University of Reading, The | 76.3 | 117 |
| 43 | Queen’s University of Belfast, The | 76.0 | 149 |
| 44 | University of Aberdeen, The | 75.9 | 84 |
| 45 | University of Strathclyde, The | 75.7 | 72 |
| 46 | Staffordshire University | 75.6 | 85 |
| 47 | University of York, The | 75.6 | 121 |
| 48 | St George’s Medical School | 75.4 | 33 |
| 49 | Southampton Solent University | 75.2 | 34 |
| 50 | University of Portsmouth, The | 75.2 | 141 |
| 51 | Queen Mary, University of London | 75.2 | 104 |
| 52 | University of Manchester | 75.1 | 221 |
| 53 | Aston University | 75.0 | 66 |
| 54 | University of Derby | 75.0 | 33 |
| 55 | University College London | 74.8 | 114 |
| 56 | Sheffield Hallam University | 74.8 | 159 |
| 57 | Glasgow Caledonian University | 74.6 | 72 |
| 58 | King’s College London | 74.6 | 101 |
| 59 | Brunel University | 74.4 | 64 |
| 60 | Heriot-Watt University | 74.1 | 35 |
| 61 | Imperial College of Science, Technology & Medicine | 73.9 | 111 |
| 62 | De Montfort University | 73.6 | 83 |
| 63 | Bath Spa University | 73.4 | 64 |
| 64 | Bournemouth University | 73.3 | 128 |
| 65 | University of the West of England, Bristol | 73.3 | 207 |
| 66 | Leeds Metropolitan University | 73.1 | 143 |
| 67 | University of Chester | 72.5 | 61 |
| 68 | University of Bristol, The | 72.3 | 145 |
| 69 | Royal Holloway, University of London | 72.1 | 59 |
| 70 | Canterbury Christ Church University | 71.8 | 78 |
| 71 | University of Huddersfield, The | 71.8 | 97 |
| 72 | York St John University College | 71.8 | 31 |
| 72 | University of Wales Institute, Cardiff | 71.8 | 41 |
| 74 | University of Glamorgan | 71.6 | 84 |
| 75 | University of Salford, The | 71.2 | 58 |
| 76 | Roehampton University | 71.1 | 47 |
| 77 | Manchester Metropolitan University, The | 71.1 | 131 |
| 78 | University of Northampton | 70.8 | 42 |
| 79 | University of Sunderland, The | 70.8 | 61 |
| 80 | Kingston University | 70.7 | 121 |
| 81 | University of Bradford, The | 70.6 | 33 |
| 82 | Oxford Brookes University | 70.5 | 99 |
| 83 | University of Ulster | 70.3 | 61 |
| 84 | Coventry University | 69.9 | 82 |
| 85 | University of Brighton, The | 69.4 | 106 |
| 86 | University of Hertfordshire | 68.9 | 138 |
| 87 | University of Bedfordshire | 68.6 | 44 |
| 88 | Queen Margaret University, Edinburgh | 68.5 | 35 |
| 89 | London School of Economics and Political Science | 68.4 | 73 |
| 90 | Royal Veterinary College, The | 68.2 | 43 |
| 91 | Anglia Ruskin University | 68.1 | 71 |
| 92 | Birmingham City University | 67.7 | 109 |
| 93 | University of Wolverhampton, The | 67.5 | 72 |
| 94 | Liverpool John Moores University | 67.2 | 103 |
| 95 | Goldsmiths College | 66.9 | 42 |
| 96 | Napier University | 65.5 | 63 |
| 97 | London South Bank University | 64.9 | 44 |
| 98 | City University | 64.6 | 44 |
| 99 | University of Greenwich, The | 63.9 | 67 |
| 100 | University of the Arts London | 62.8 | 40 |
| 101 | Middlesex University | 61.4 | 51 |
| 102 | University of Westminster, The | 60.4 | 76 |
| 103 | London Metropolitan University | 55.2 | 37 |
| 104 | University of East London, The | 54.2 | 41 |
| 10465 |
The maximum overall score is 100 and the figure in the rightmost column is the number of students from that particular University that contributed to the survey. The total number of students involved is shown at the bottom, i.e. 10465.
My current employer, Cardiff University, comes out pretty well (17th) in this league table, but some do surprisingly poorly such as Imperial which is 61st. No doubt University spin doctors around the country will be working themselves into a frenzy trying how best to present their showing in the list, but before they get too carried away I want to dampen their enthusiasm.
Let’s take Cardiff as an example. The number of students whose responses produced the score of 79.4 was just 180. That’s by no means the smallest sample in the survey, either. Cardiff University has approximately 20,000 undergraduates. The score in this table is therefore obtained from less than 1% of the relevant student population. How representative can the results be, given that the sample is so incredibly small?
What is conspicuous by its absence from this table is any measure of the “margin-of-error” of the estimated score. What I mean by this is how much the sample score would change for Cardiff if a different set of 180 students were involved. Unless every Cardiff student gives Cardiff exactly 79.4 then the score will vary from sample to sample. The smaller the sample, the larger the resulting uncertainty.
Given a survey of this type it should be quite straightforward to calculate the spread of scores from student to student within a sample from a given University in terms of the standard deviation, σ, as well as the mean score. Unfortunately, this survey does not include this information. However, lets suppose for the sake of argument that the standard deviation for Cardiff is quite small, say 10% of the mean value, i.e. 7.94. I imagine that it’s much larger than that, in fact, but this is just meant to be by way of an illustration.
If you have a sample size of N then the standard error of the mean is going to be roughly (σ⁄√N) which, for Cardiff, is about 0.6. Assuming everything has a normal distribution, this would mean that the “true” score for the full population of Cardiff students has a 95% chance of being within two standard errors of the mean, i.e. between 78.2 and 80.6. This means Cardiff could really be as high as 9th place or as low as 23rd, and that’s making very conservative assumptions about how much one student differs from another within each institution.
That example is just for illustration, and the figures may well be wrong, but my main gripe is that I don’t understand how these guys can get away with publishing results like this without listing the margin of error at all. Perhaps its because that would make it obvious how unreliable the rankings are? Whatever the reason we’d never get away with publishing results without errors in a serious scientific journal.
Still, at least there’s been one improvement since last year: the 2009 results gave every score to two decimal places! My A-level physics teacher would have torn strips off me if I’d done that!
Precision, you see, is not the same as accuracy….
In the Dark 
January 15, 2010 at 1:05 pm
These results have as much impact as a table entitled “Universities most proficient in persuading students to fill out b******t questionnaires”
January 15, 2010 at 1:05 pm
-Sadly it would seem my peers have little else to do….
January 16, 2010 at 11:23 am
All the same, the results do conform to my idea about how university league tables are compiled: obtain some statistics about universities, produce some index using some arbitrary combination of the data, rank the universities, then reject the index if Oxford and Cambridge do not come within the top few and find some new way of combining the data to form an index, then continue until Oxbridge finally comes top.
January 16, 2010 at 11:31 am
Bryn,
Oxbridge isn’t top of this one. The best university in the country is actually Loughborough (pronounced “Lowbrow”).
Peter
January 17, 2010 at 2:11 pm
Thanks for the explanation. Every year the University I work at bust a gut to try to get students to fill out the national student survey, on the assumption that students they’ve badgered into completing it will give a positive impression of the place.
I understand the desire to know what students’ opinons of institutions are, but the tiny samples are pretty meaningless. Yet the effort and resources poured into it continue.
January 17, 2010 at 2:34 pm
Maybe in a few years time only a handful of students will be going to University, so such surveys will be more representative.
The National Student Survey (NSS) is a different thing, actually. The one I was talking about was run by the Times Higher itself. The NSS attempts to survey all final-year undergraduates at each institution, but since it is voluntary there is a self-selection of respondents which probably introduces some sort of bias.
I’ve got nothing against surveys of student opinion, or anything else for that matter, but it really annoys me when they are published without any discussion of their reliability. If such things are going to be published they should be accompanied by a clear warning about their reliability. Unfortunately, politicians, civil servants and, especially, University administrators are largely incapable of understanding arguments involving statistics so companies continue to peddle this twaddle and people lap it up.