This site now Cloudflare-free

About a year ago I set up Cloudflare as a fronting CDN for this site and my music site because it was the most expedient way of dealing with an AI bot onslaught. It helped a bit but the bots very quickly figured out how to get around all that and while Cloudflare gave me some slightly-better management tools for some stuff, I figured out better approaches to the bot mitigation.

Cloudflare was also super aggressive about caching some stuff that I didn’t want to be cached, and of course, there are many, many political and ideological reasons to not want to use Cloudflare. So my plan was always to switch back to not being under Cloudflare, but the longer I waited the harder it seemed like it would be, due to how SSL certificates work. In particular, I use wildcard Let’s Encrypt certificates, which require DNS to be current, and a big thing that Cloudflare does is… take over your DNS.

But tonight I got a hair up my butt and switched back to my own termination, and it wasn’t too hard to do, with just a little bit of DNS and TLS juggling, and I wanted to minimize my website downtime.

Read more…

Fuck AI LLM scrapers

Wellp, my whack-a-mole approach finally got to be too much to maintain. The last day or so my server has been absolutely inundated with traffic from thousands of IP blocks, all coming from China, and I got sick of trying to keep up with it myself.

I looked into setting up Anubis and preparing to just whitelist a lot of IndieWeb things, but it’s all just so very overwhelming and for now I’ve gone with Cloudflare, problematic as they are, because the amount of energy I can put into this shrinks every day and sometimes I just want things to stop sucking for a while.

All of my DNS has propagated but of course it’ll be a while before the bots decide to update their own DNS caches, so my server is still getting absolutely hammered, but hopefully things will subside, and in the meantime things are at least responsive.

I guess at some point I’ll have to figure out how to actually set up TLS with Cloudflare (since I’ve been using Letsencrypt wildcard certs but obviously those don’t work anymore when Cloudflare is handling my DNS) but that’s a problem for future me. Also I’ll definitely be on the lookout to make sure that Cloudflare is properly honoring my login cookies. It’d definitely be unfortunate if it gets confused about logins, which is one of the more common failure modes with HTTP proxies.

I’m also super worried that this will interfere with IndieWeb stuff, because of course most of the anti-bot things assume that any traffic coming from data centers or from headless/scriptless user agents is abusive. Which is, y'know, 99.99% accurate, but that 0.01% is stuff I really care about (namely interop).

Anyway. I resent that this is the state of the Internet right now. It’s getting really difficult for me to find anything positive about AI when this is how the industry treats everyone.