Abubis - Staying Online with AI Bots

From EPrints Documentation
Revision as of 13:19, 12 October 2026 by LukeWallin (talk | contribs) (→‎Our Solution)
Jump to: navigation, search

Hardening your EPrints instance so it can survive the onslaught of bizarre behaviour from (presumed) AI bots.

From 2025(ish) onwards, EPrints instances can struggle to stay online from DDOS-like behaviour that is flung at it regularly.

EPrints Services have been working on a solution that uses Anubis to attempt filter out bots for specific pages, and nginx to add a cache and apply rate and connection limiting. Our solution is tailored to our hosting infrastructure, but should be relatively easy to adapt to a bespoke EPrints installation. This page documents how the solution works and how it is configured.

Our Solution

The ingredient "Anubis" contains our solution with Anubis and Nginx. Its README will be the most up to date source of information, therefore this will only summarise.

  • EPrints is run on its own internal instance of Apache, because it is unstable in Apache's multithreaded mode, and the layers in front exacerbate the effects of a segfault.
  • Apache is used as the front webserver (listening on port 80/443) because this is currently the only way to support Shibboleth Single Sign On and SSL is already configured for Apache.
  • Nginx is used instead of Apache because it natively supports more open connections than Apache (originally it was intended to be the front facing webserver) and it has some very useful rate limiting and caching features.
  • In order to support not caching logged in users, the ingredient has some fixes for core EPrints code for setting and unsetting session cookies. Hopefully the next (3.4.9) release of EPrints will include these fixes.
Retrieved from ‘https://wiki.eprints.org/w/index.php?title=Abubis_-_Staying_Online_with_AI_Bots&oldid=17550’