EDIT: you don't need to update ssh now

as an update, the general AI-training background of the internet is now impossible to tell apart from being under attack

akk dev was being hit by enough requests to overwhelm our CPU, so we've had to put in some protections

if you use SSH clones of our repositories, the domain is now `ssh.akkoma.dev` - you'll need to update your remote urls - sorry for the inconvenience

@akkoma yuck yuck yuck yuck clownflare

@hsza i know, but what are you gonna do? i was already using anubis, but the volume of traffic is so high that the CPU literally cannot generate enough challenges for them

@akkoma would it be possible to give me a bypass cookie (preferably putting it somewhere like the irc channel topic where scrapers won't find it but other people can)? I can't get past the cloudflare challenge otherwise

also irc seems to be down right now

@akkoma iocaine!! cc @algernon

not that it’s too relevant now, but heres an article i saw linked semi-recently about how anubis is pretty unhelpful https://hackerdude.tech/vault/Anubis/

@hsza @akkoma would iocaine not use even more cpu than anubis tho, because it has to actually generate data in a markov chain

@nyx @akkoma iocaine is comparatively quite lightweight actually fwik

@hsza @nyx if you're interested, this is the level of traffic it would have to handle
without throwing more CPU at the problem, there are few solutions other than offloading it to a third party
cringe it is, but given that nobody here is a sysadmin, it's what we've got

@nyx @hsza @akkoma iocaine's markov generator is fast. Like, it can generate a ~6k page of garbage faster than you serve a static file from SSD (unless it's already in the fs cache).

When it comes to iocaine, the bottleneck is not going to be the garbage generation. A TLS handshakes is about ten times more expensive.

What iocaine can do, is poison the crawler's URL queue, and fill it with identifiable URLs. When they come back hitting those, plonk their IPs into the firewall to block them for a little while. That's a built-in feature, you don't need to fail2ban or anything, just a bit of configuration.

This decreased my incoming requests from ~100million/day to ~2.5million/day, and a €8/month VPS now happily fronts a few dozen domains, including my forge. iocaine's sitting at... around 2-5% CPU (on a 2vCPU server) for most of the day.

So, no. It would not use more CPU. It was built to run on a potato.

@akkoma @nyx i do think @algernon would be able to help out ^^

@hsza @algernon @nyx hey this is pretty cool honestly
generating complete nonsense with silly links is very funny

@hsza @algernon @nyx wait this is REALLY funny

@akkoma @nyx @hsza If you like that, check out this awesomeness @lukasrotermund made of it.

@algernon @nyx @hsza @lukasrotermund this whole thing is really cool like the reason i didn’t expect to be able to solve this without 3rd party intervention is problems like “lmao scrapers just use standard browser UAs” and such having rules for that like in nam-shub genuinely might make cloudflare not as required

for fun, basic metricys

# TYPE nam_shub_of_enki_garbage_generated counter
nam_shub_of_enki_garbage_generated{host="akkoma.dev"} 7404077
# HELP nam_shub_of_enki_request_distribution Request distribution (per group and outcome)
# TYPE nam_shub_of_enki_request_distribution counter
nam_shub_of_enki_request_distribution{group="Aggressive maze explorers",outcome="garbage"} 6
nam_shub_of_enki_request_distribution{group="Cookie Monster food",outcome="challenge"} 1288
nam_shub_of_enki_request_distribution{group="Disguised bots",outcome="garbage"} 544
nam_shub_of_enki_request_distribution{group="Facebook",outcome="garbage"} 6
nam_shub_of_enki_request_distribution{group="Google",outcome="garbage"} 227
nam_shub_of_enki_request_distribution{group="Other",outcome="garbage"} 2166
nam_shub_of_enki_request_distribution{group="Other",outcome="not-for-us"} 108
nam_shub_of_enki_request_distribution{group="fediverse-software",outcome="challenge"} 46

mmmm they are being eaten…

alright turns out there's a better way
i'll cname ssh over to root so if you changed it, no harm no foul, but i think we can revert to the old setup with some coolio interventions

@akkoma
When I access akkoma.dev using the Vivaldi browser installed via Flatpak on Debian 13, I get flagged as an AI scraper. I have AdGuard installed as an extension, but disabling it didn’t change the outcome.
I can access the site from Floorp on the same PC, however.

@pppppppppoe this is expected and, i am reasonably sure not possible to mitigate easily - vivaldi uses a faked user-agent string with mismatched fields, which is the same way bots try to blend in and evade blocks

you may find some success in updating the browser, or finding a UA string that matches your platform

see https://vivaldi.com/blog/user-agent-changes/