Just very high usage all of a sudden, after years of reasonable usage. Google has indexed it (respectfully) since 2008 just fine.
New traffic isn't humans. I blocked some AI scraper user-agents, which helped, a bit. But most new user agents are identifying as vanilla browsers, not scrapers.
I don't have numbers. It was enough to consume all nginx worker_connections. Raising the number doesn't help, as it's just reverse proxying to JVM.
After the switch, Cloudflare showed USA and Singapore as heavy traffic sources.
I don't mind scrapers on the site, but app is a search engine (of sorts) so every page view consumes some CPU. Including 'facet this search' buttons. My (WIP) solution is to rewrite to make it all client-side and put it all on a CDN.
alt Hacker News
Just very high usage all of a sudden, after years of reasonable usage. Google has indexed it (respectfully) since 2008 just fine.
New traffic isn't humans. I blocked some AI scraper user-agents, which helped, a bit. But most new user agents are identifying as vanilla browsers, not scrapers.
I don't have numbers. It was enough to consume all nginx worker_connections. Raising the number doesn't help, as it's just reverse proxying to JVM.
After the switch, Cloudflare showed USA and Singapore as heavy traffic sources.
I don't mind scrapers on the site, but app is a search engine (of sorts) so every page view consumes some CPU. Including 'facet this search' buttons. My (WIP) solution is to rewrite to make it all client-side and put it all on a CDN.