In the Dark

There’s an interesting paper on arXiv with the title On the Use of Letters of Recommendation in Astronomy and Astrophysics Graduate Admissions and the abstract

Letters of recommendation are a common tool used in graduate admissions. Most admissions systems require three letters for each applicant, burdening both letter writers and admissions committees with a heavy work load that may not be time well-spent. Most applicants do not have three research advisors who can comment meaningfully on research readiness, adding a large number of letters that are not useful. Ideally, letters of recommendation will showcase the students’ promise for a research career, but in practice, the letters often do not fulfill this purpose. As a group of early and mid-career faculty who write dozens of letters every year for promising undergraduates, we are concerned and overburdened by the inefficiencies of the current system. In this open letter to the AAS Graduate Admissions Task Force, we offer an alternative to the current use of letters of recommendation: a portfolio submitted by the student, which highlights e.g., a paper, plot, or presentation that represents their past work and readiness for grad school, uploaded to a centralized system used by astronomy and astrophysics PhD programs. While we argue that we could eliminate letters in this new paradigm, it may instead be advisable to limit the number of letters of recommendation to one per applicant.

Barron et al, arXiv:2412.0871

This reminds me of an old post (from 2009) on the topic of recommendation letters or testimonials that proved quite controversial at the time. I’ll rehash part of it now because my views have changed, though the situation is similar to the UK where I was based when I wrote the original post.

In my view, the role a reference letter should be as factual as possible, and probably the most important thing it contains is confirmation that the information given by a student in their application is accurate. This could be done in a simple pro forma, and referees are often asked to complete such things nowadays. I think this is reasonable, but the questionnaires concerned are frequently so poorly designed as to be useless.

The principal bone of contention with my earlier post was whether a Professor should ever write critical or even negative comments when asked to recommend a student for a place on a graduate course. In most of my career I haven’t really thought of these letters as much “recommendations” as “references” or “testimonials” which are supposed to describe the candidate’s character and abilities in a manner that is useful to those doing the recruitment. They are not meant to be written in absurdly hyperbolic terms nor are they meant to ignore any demonstrable shortcomings of the applicant. They are supposed to advise the people doing the recruitment of the suitability of the candidate in a sober, balanced and objective way. Fortunately, most students applying to graduate schools are actually very good so there are many more positives than negatives, but if there are weaknesses in my view these must be mentioned. Hype should not be involved. The point is that the referee is not only providing a service for the student but also for the recruiting school. On this basis, it is, I think, perfectly valid to include negative points as long as they can be justified objectively. I – and I’m sure others on this side of the pond – have been criticized by our transatlantic colleagues for writing very reserved recommendation letters, but having one year received references from a US institution on behalf of 4 different students all of whom were apparently the best student that institution had ever had in physics, I think I prefer the understated style.

However, references transcripts and other paperwork can only establish whether a student has reached the threshold level of technical competence that is needed to commence a research degree. That’s a necessary but not sufficient condition for their success as a scientist. The other factors – drive, imagination, commitment, diligence, etc – are much harder to assess. I think this part has to be done at interview. You can’t just rely on examination results because it’s by no means true that the best students at passing examinations necessarily evolve into the best graduate students.

A big change in the 15 years since I wrote my original post is that undergraduate programmes now often include some form of research project and students often have access to internships of various kinds. The performance of a student on such programmes is clearly important in determining their likely performance as a graduate student, so comments on these could be invaluable to a selection committee.

To respond to the paper above, therefore, I would say there is a case for reducing the number of reference letters to one, factual letter, and to base most of the selection on interviews. This would I think make the system fairer, but would not reduce workload as the interviews would take longer to organize and carry out.