The results of the UK’s Research Excellence Framework (REF) 2021 have now been published. You can find them all here at the REF’s own website because they are presented there in a much more informative way than the half-baked “rankings” favoured by, e.g., The Times Higher.
To give some background: the overall REF score for a Unit of Assessment (UoA; usually a Department or School) is obtained by adding three different components: outputs (quality of research papers); impact (referring to the impact beyond academia); and environment (which measures such things as grant income, numbers of PhD students and general infrastructure). Scores are assigned to these categories, e.g. for submitted outputs on a scale of 4* (world-leading), 3* (internationally excellent), 2* (internationally recognized), 1* (nationally recognized) and unclassified. Similar star ratings are applied to the impact and environment. These are weighted at 60%, 25% and 15% respectively in the current incarnation of REF.
You can find further discussion of the REF submission rules, especially concerning changes with respect to 2014, here.
The way the star ratings are often reported is via a Grade Point Average reflecting the percentage of in each band. A hypothetical UoA that scored 100% in the top category would have a GPA of 4.0, for example. One that had 50% 4* and 50* 3* would be 3.5, and so on.
In the 2014 REF institutions were allowed to be selective in the number of staff submitted so the GPA wasn’t really a very appropriate measure: some institutions chose to submit only their very best research in order to get a high GPA. The funding allocated as a result of REF turned out to be highly weighted towards 4* so this was a sensible strategy for them, but it made the simple GPA-based rankings even more meaningless than usual. That didn’t stop e.g. The Times Higher making such rankings though.
This time the rules on selection are stricter so the GPA is arguably more relevant, though many institutions have achieved selectivity anyway by moving certain staff onto teaching-only contracts. Staff on such contracts do not have to be submitted. I note that the main REF website does not use the GPA at all but instead gives profiles like this:
I show the example of Sussex because of my bad memories of the last REF (the 2014 exercise). I had moved to Sussex in 2013 at which point preparations were not well advanced and although everyone concerned worked very hard to put together the best submission for Physics & Astronomy we had to face the problem that our staff numbers had grown significantly in 2013 response to an increase in student numbers. While new staff could bring publications with them they couldn’t bring impact or environment, and while the outputs scored well the latter two categories didn’t so the Department of Physics & Astronomy did poorly in the ensuing rankings.
It must be said however that the primary purpose of the REF (allegedly) is to allocate blocks of funding- the so-called QR funding – to support research in the UoA concerned and while the GPA at Sussex was disappointing the fact that the money depends on the number of staff submitted meant that we got a substantial increase in QR dosh. Note further that the formula for allocation of funds to 4*, 3*, etc is not even specified in advance of the exercise: it is likely to be highly concentrated on research graded 4* and that the funding formula will probably different in England, Wales, Scotland and Northern Ireland. A ranking in terms of money earned is likely to look rather different from one based on GPA.
Another, even more fundamental, problem with the GPA is that the scores are so close together that the differences are of doubtful significance. In the Physics UoA, for example, the gap between top GPA (Sheffield) and 5th place (Bristol) is just 0.05 (3.65 versus 3.60) respectively. I see also that Cardiff is ranked equal 18th (with Imperial College) on a GPA of 3.45.
I say these things just to illustrate how much more subtle the criteria for success are compared to a simple GPA. It’s even hard to tell on an objective basis who to congratulate and who to commiserate.
Anyway, back to Sussex I see that Physics & Astronomy has done far better on environment and impact than last time round and the outputs (95% of which are either 4* or 3*) are comparable to last time (96%) so by those measures they have done well although this might not be reflected in a GPA-based ranking. Sussex is 26th in the rankings with a GPA of 3.35, if you’re interested, which is better than last time, though they will probably be disappointed at the presence of 2* elements in their profile.
Indeed, looking through the Physics list I can’t see any UoA that has a lower GPA this time than last time. The pot of money to be allocated for QR funding is fixed so if every UoA does better that doesn’t mean every UoA gets more money; some institutions will no doubt find that their improved GPA is accompanied by a cut in QR funding.
I’ll end by re-iterating that, having moved to Ireland in 2017, I’m very glad to be out of the path of the bureaucratic juggernaut that is the REF. In its first incarnations (as the Research Assessment Exercise) it did fulfil a useful purpose and did, I believe, improve the quality of UK research. Since then, however, it has become an industry that is largely self-serving. I quote from an article in the Times Higher itself:
The allocation of QR funding could be done in a much simpler and fairer way but the REF is now such a huge edifice it will resist being replaced by something smaller. No doubt before long the staff who spent so much time preparing for REF 2021 will start work on the next exercise. And so it goes on.
The changes in ranking that now occur from exercise to exercise are generally small in magnitude and in number. In other words, huge effort and cost are being invested to discover less and less information.
P.S. For completeness I should say that I am glad we don’t have an equivalent of the REF here in Ireland, we don’t have an equivalent of the QR funding either. This latter is a serious problem for the sustainability of research in third-level institutions, and it is not addressed at all in the recent proposals for reform.
