In the Dark

A few weeks ago I mentioned the concerning news that arXiv was changing the way it works and moving all of its content into cloud storage. Related to this was a decision made last year to shut down the previously existing arXiv mirror sites. At the time arXiv explained that

The arXiv mirror network served a role – acting as a backup for the corpus, allowing some degree of load distribution, and providing improved access for users who were geographically closer to a mirror – that is no longer necessary. arXiv now has multiple backups for the arXiv corpus in place, and the Fastly CDN (Content Delivery Network) that we use to deliver content provides excellent service throughout the world.

This decision, which puts all the eggs in one basket, is looking very questionable after in the Trump era. The already oppressive restrictions on academic freedom in the United States are expected to escalate further. These developments will affect research infrastructures worldwide. In other words, the USA has become a single-point failure. This ongoing and escalating risk can only be mitigated by moving to a more decentralized and thus more resilient infrastructure.

One move in this direction has been made by the German National Library for Science and Technology which, in German, is the Technische Informationsbibliothek or TIB for short; their website is here. As explained here, TIB is in the process of creating a “dark archive” of the arXiv, i.e. a backup of all the arXiv content. According to TIB,

The establishment of a “dark archive” is an expression of our long-standing commitment to reliable, international scientific provision and as a partner of arXiv. Even though the “dark archive” currently only operates in the background, it is a crucial building block for the long-term safeguarding of digital research content, because in the event of a crisis, we can open the archive.

In other words, there will be a backup that can be activated if the arXiv main site collapses.

I think this is a valuable precaution, and there should probably be more dark mirrors of this kind around the world. As well as this specific measure I also endorse the general philosophy of creating a “more decentralized and thus more resilient infrastructure”. Yesterday I did an interview with a journalist about the Open Journal of Astrophysics at the end of which I said that I thought the future of academic publishing was a federated system of overlays over a wide range of institutional and/or subject repositories. That’s the only way to spread the cost of maintaining the infrastructure in a reasonable way as well as reducing the clear vulnerability of the current system.