A Peer Review Poll
A long time ago I posted a poll to see what people think about the issue of peer review. Now seems a good time to circulate it again.
In previous posts (e.g. this one) I had advanced the view that, at least in the subject I work in (astrophysics), while in its usual form peer review does achieve some degree of quality control, it is by no means perfect. Some good papers get rejected and some poor papers get accepted. Moreover, the refereeing is usually done for free by members of the academic community while journal publishers use peer review as a justification for levying publication charges in that it provides added value to the publication process – a view I disputed here.
Any system operated by humans is bound to be flawed to some extent, but the question is whether there might be a way to improve the system so that it is fairer and more transparent.
I suggested that it could be replaced by a kind of crowd-sourcing, in which papers are put on an open-access archive or repository of some sort, and can then be commented upon by the community and from which they can be cited by other researchers. This would, if you like, be a sort of “arXiv plus” – good papers would attract attention and poor ones would disappear.
We did consider having open peer review of the sort mentioned above for the Open Journal of Astrophysics but this option was not available for the no-frills off-the-shelf Scholastica platform we went for so we now operate a version of the traditional peer review system. This achieves some level of gate-keeping but also (and much more importantly, in my view) makes constructive criticism to allow authors to improve their papers. We also discussed publishing referee reports alongside the papers, but that is also beyond the scope of our current system (and would of course require the consent of referees).
I have no idea really how strongly others rate the current system of peer review. The following poll is not very scientific, but ‘ve tried to include a reasonably representative range of views from “everything’s OK – let’s keep the current system” to the radical suggestion I make above.
Follow @telescoper
In the Dark 
February 14, 2022 at 8:48 pm
Depends who your peers are…
February 15, 2022 at 3:25 am
The Peer Community in (PCI) initiative comes close to these ideals, from having open reviews (anonymous or not) to bringing in PCI-friendly journals that take the earlier reviews in account. Despite their shortcomings, machine learning conferences also share some features with the same goal, by publishing (anonymous) reviews and authors’ replies, asking for code, and linking reviews from previous conferences when the paoer is resubmitted.
February 15, 2022 at 9:34 pm
Depends which journal. MNRAS yes, Nature no. Also it’s often forgotten in these discussions that arXiv isn’t a free publishing platform. It looks free for academic users, but it has real costs, paid for by institutions, sponsors etc. OJA is a great idea, and it’s wonderful to see it succeed – the subtlety being that some costs are shifted elsewhere. This is fine, but it’s something the academy will need to keep an eye on if the arXiv-top-end journals grow, to make sure arXiv continues to be sustainably funded. My personal view is that ultimately it’s the academy (including learning societies) who should be running at least the majority of the publishing infrastructure.
February 16, 2022 at 8:21 am
I agree but remember the arXiv runs at a fraction of the cost of global library subscriptions to journals. I would like to see institutions divert funding from funding journals to making repositories such as arXiv more sustainable.
February 16, 2022 at 8:25 am
I would love it if eg the RAS would take over and run OJAp as a free open access journal, but that would have to be treated in a different way from its existent journals ie as a (small) cost rather than a way of generating income.
February 16, 2022 at 1:57 pm
Interesting thought Peter re OJAp and a future role for the RAS. Personally I don’t mind journals being income-generating, provided (a) they’re not profit-generating, and (b) revenues are ploughed back into the academy. For example the RAS journals generate income for the RAS, but the RAS is a charity so it’s legally forbidden to make a profit, so the revenue gets fed back to the community to fund fellowships, meetings, policy influencing etc. At the moment the RAS don’t have capacity to run the publishing arm themselves so they have to contract out to eg Wiley or more recently OUP. As OJAp blazes a trail for running a low cost publishing operation, maybe as a minimum the RAS will have more options in future. But “it’s tough to make predictions, especially about the future”
February 16, 2022 at 2:15 pm
Well, OUP takes a big slice for profit too. I don’t mind the RAS raising money (of course) but my complaint is that it actually comes out of library budgets.
February 16, 2022 at 2:34 pm
Just saw this on Twitter… https://twitter.com/dglaucomflecken/status/1493946227985059841?s=21