In the Dark

Dear Rankers,

I note with interest that you have announced significant changes to the methodology deployed in the construction of this years forthcoming league tables. I would like to ask what steps you will take to make it clear to that any changes in institutional “performance” (whatever that is supposed to mean) could well be explained simply by changes in the metrics and how they are combined?,

I assume, as intelligent and responsible people, that you did the obvious test for this effect, i.e. to construct and publish a parallel set of league tables, with this year’s input data but last year’s methodology, which would make it easy to isolate changes in methodology from changes in the performance indicators.  This is a simple test that anyone with any scientific training would perform.

You have not done this on any of the previous occasions on which you have introduced changes in methodology. Perhaps this lamentable failure of process was the result of multiple oversights. Had you deliberately withheld evidence of the unreliability of your conclusions you would have left yourselves open to an accusation of gross dishonesty, which I am sure would be unfair.

Happily, however, there is a very easy way to allay the fears of the global university community that the world rankings are being manipulated. All you need to do is publish a set of league tables using the 2022 methodology and the 2023 data. Any difference between this table and the one you published would then simply be an artefact and the new ranking can be ignored.

I’m sure you are as anxious as anyone else to prove that the changes this year are not simply artificially-induced “churn”, and I look forward to seeing the results of this straightforward calculation published in the Times Higher as soon as possible, preferably next week when you announce this years league tables.

I look forward to seeing your response to the above through the comments box, or elsewhere. As long as you fail to provide a calibration of the sort I have described, this year’s league tables will be even more meaningless than usual. Still, at least the Times Higher provides you with a platform from which you can apologize to the global academic community for wasting their time and that of others.

This entry was posted on September 20, 2023 at 10:12 am and is filed under Bad Statistics, Education with tags Churn, League Tables, Rank, Rankers, Ranking, Times Higher Education, World University Rankings. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.