alt Hacker News> Most users aren't spoofing their user agent headers to be a different operating system.
The people behind the LLM behind this blog post are. They're trying to pretend their robots are people to sell other websites' data to their customer. It's easier to pass bot detection gates if you pretend to be a physical machine running Windows or macOS than if you honestly admit you're using Linux on a VM.
Replies
Some scrape activity happens because I can't obtain the data any other way. I would be thrilled if certain retailers had price and availability data as an API so I could not bother with the bulk of scrape and process.
The HTTP User-Agent header was a mistake from the beginning. There is no legitimate need for the server to know what software the client is (or claims to be) running.
It's sometimes easier to lie than to tell the truth, and being on Linux telling the truth gets me more scrutiny than those pretending to be legit.