> So the question is: can you write a web scraper in your browser? The answer is: YES, you can! So why is nobody doing it?
> One of the issues is what is called CORS (Cross-Origin Resource Sharing) which is a set of protocols which may forbid or allow access to a web resource by Javascript. There are two possible workarounds: a browser extension or a proxy server. The first choice is fairly limited since some security restrictions still apply.
I'm doing this for a browser extension that crawls a website from page to page checking for SEO/speed/security problems (https://www.checkbot.io/). It's been flexible enough, and it's nice not to have to maintain and scale servers for the web crawling. https://browserflow.app/ is another extension I know of that does scraping within the browser I think, and other automation.
Interesting, I’ve tried checkbot recently and it failed to do any cloudflare gated website, even 1 page. But maybe I’m on the old version