Never mind the points, look at the line!
I was just thinking this morning that it’s been a while since I posted anything in my Bad Statistics folder when suddenly I come across this gem from a paper in Nature Astronomy entitled Could quantum gravity slow down neutrinos?
The paper itself is behind a paywall (though a preprint version is on the arXiv here). The results in the paper were deemed so important that Nature Astronomy tweeted about them, including this remarkable graph:
Understandably there has been quite a lot of reaction from scientists on Twitter to this plot, questioning how the blue line is obtained from the dots (as only one point to the right appears to be responsible for the trend), remarking on the complete absence of any error bars on either axis for any of the points, and above all wondering how this managed to get past a referee, never mind one for a “prestigious” journal such as Nature Astronomy. It wouldn’t have passed muster as an undergraduate exercise.
Of course this is how a proper astronomer would do it:
Joking aside, if you look at the paper (or the preprint if you can’t afford it) you will see another graph, which shows two other points at higher energy (red triangles):
The extra two points don’t have any error-bars either, and according to the preprint these appear to be unconfirmed candidate GRB events.
The abstract of the paper is:
In addition to its implications for astrophysics, the hunt for neutrinos originating from gamma-ray bursts could also be significant in quantum-gravity research, as they are excellent probes of the microscopic fabric of spacetime. Some previous studies based on neutrinos observed by the IceCube observatory found intriguing preliminary evidence that some of them might be gamma-ray burst neutrinos whose travel times are affected by quantum properties of spacetime that would slow down some of the neutrinos while speeding up others. The IceCube collaboration recently significantly revised the estimates of the direction of observation of their neutrinos, and we here investigate how the corrected directional information affects the results of the previous quantum-spacetime-inspired analyses. We find that there is now little evidence for neutrinos being sped up by quantum spacetime properties, whereas the evidence for neutrinos being slowed down by quantum spacetime is even stronger than previously determined. Our most conservative estimates find a false-alarm probability of less than 1% for these ‘slow neutrinos’, providing motivation for future studies on larger data samples.
I agree with the last sentence where it says larger data samples are needed in future, but also I’d suggest higher standards of data analysis are also called for. Not to mention refereeing. After all, it’s the quality of the reviewing that you pay for, isn’t it?
P.S. For those of you wondering, this paper would not have been published by the Open Journal of Astrophysics even if passed review, as it is not on the astro-ph section of arXiv (it’s on gr-qc).
Follow @telescoper
In the Dark 
June 14, 2023 at 2:55 pm
I would have been laughed at if I’d used a plot like that in any of my Astronomy practicals at UCL
June 14, 2023 at 3:36 pm
I’m surprised an entire collaboration signed off on this! This is from the whole of Icecube? Or I misunderstood and third parties have used public data?
June 14, 2023 at 5:08 pm
There are only five authors on the paper…
June 14, 2023 at 3:03 pm
“Clearly…”
Does OJA limit itself to arXiv papers in the astro-ph section? There is a grey area between that and gr-qc which you are missing out on in that case.
June 14, 2023 at 3:06 pm
Yes it does, but we can accept papers that are cross-listed which most in the grey area are. There are moves to set up an overlay journal for gr-qc anyway.
June 15, 2023 at 12:38 am
Using new directional data for neutrinos, an attempt is made to associate particular neutrinos with particular GRBs. If these associations are correct, then timing differences (y coord) between em and neutrinos are accurate.
The x coordinate of the data points depends on neutrino energy and on GRB redshift. GRB redshift was measured directly for only one data point. The redshifts for the other 6 points were estimated based on the the one known redshift.
The authors’ analysis predicts an 83% chance of a 1 wrong neutrino/GRB association and 39% chance of at least 2 wrong associations, i.e., completely wrong y coords (and, I think x coords). This, for a sample of 7 data points!
Fig 2 adds 2 data points for higher energy neutrinos. These would have em/neutrino time differences so long that picking out one particular GRB is too challenging. Consequently, to add these extra 2 points, the inferred Fig 1 line slope is used to identify candidates!
June 15, 2023 at 12:34 pm
“There are only five authors on the paper…”
Ah ok, I see. Looking at their other papers all five appear to have a theoretical background.
It’s curious that the reviewer is a member of Icecube. Members of my collaboration generally worry about this sort of paper being published with our open data (and some even argued against publishing the data for this reason IIRC).
June 15, 2023 at 12:37 pm
Yes, some might consider that referee to have a conflict of interest…