Yes: I find it surprising that this isn't a more widespread approach. It's how I've taught web scraping to my PhD students for some years.
https://github.com/jawj/web-scraping-for-researchers
It’s not widespread because it’s much more complicated than making an http request and reading the results from the body. You don’t spin up a browser, much less the full GUI, unless it’s a last resort.
It’s not widespread because it’s much more complicated than making an http request and reading the results from the body. You don’t spin up a browser, much less the full GUI, unless it’s a last resort.