busybee fluffy rambles

A couple days ago, the fucking crawlers ramped up their efforts to get past my 429 error page, so I finally set up a fail2ban rule that would block anything that kept on poking at things.

Hoo boy that was the wrong thing to do. When the bot farms started to detect that I was blocking them at the IP level, they started a massive onslaught coming from massive coordinated botnets that came from everywhere, because apparently the bot networks have decided that if they’re being blocked, they’re going to do whatever they can to get around the block, and they were specifically only making a single request from every one of the tens of thousands of IP addresses.

After several hours of frustration I decided to take this site down for a bit, and found that as soon as my server started responding with error 502 making it look like it was down for the count, the botnet slowed to a trickle. Which tells me that they don’t give a shit about polite requests to please slow down, but will stop when they think that the server has been overwhelmed and taken down. But of course as soon as I brought it back up and a single bot saw that my site was back up, so returned the torrent of bullshit.

So, I changed my anticrawl thing a bit where now a failed check sends an inappropriate error 502.

As a test I did see if I could put my existing automated sentience check on the error page and the bots were still trying to submit the form, so I’ve changed the form to be manual, and to require filling in the world’s simplest (and hopefully most accessible) CAPTCHA.

Repeated failures to get past the CAPTCHA will still result in an IP block, but for now it seems that this is keeping the bots at bay.

Anyway, as soon as I instituted that change on this site, they started hammering my music site instead, so I put the same mitigations in place there too, and now my server is out of crisis mode.

I hate that I have to spend so much fucking time on this fucking bullshit, and that the solution requires violating HTTP standards. Because that seems to be all that these fuckers understand.

I’m tired, y'all.