Abubis - Staying Online with AI Bots

Hardening your EPrints instance so it can survive the onslaught of bizarre behaviour from (presumed) AI bots.

From 2025(ish) onwards, EPrints instances can struggle to stay online from DDOS-like behaviour that is flung at it regularly.

EPrints Services have been working on a solution that uses Anubis to attempt filter out bots for specific pages, and nginx to add a cache and apply rate and connection limiting. Our solution is tailored to our hosting infrastructure, but should be relatively easy to adapt to a bespoke EPrints installation. This page documents how the solution works and how it is configured.

Our Solution

The ingredient "Anubis" contains our solution with Anubis and Nginx. Its README will be the most up to date source of information, therefore this will only summarise.

• EPrints is run on its own internal instance of Apache, because it is unstable in Apache's multithreaded mode, and the layers in front exacerbate the effects of a segfault.
• Apache is used as the front webserver (listening on port 80/443) because this is currently the only way to support Shibboleth Single Sign On and SSL is already configured for Apache.