alt Hacker Newsgoogle.com/goto: Google's anti-scraping update
Comments
I did an interview with Google around 20 years ago, where they posed a challenge involving tracking which specific search results people click. It's obvious in hindsight the solution required rewriting all the urls to redirect through their servers. Note this was in the days before they already did so as a matter of course.
I failed to gain traction on the problem, because to me the very idea of doing such a thing was too reprehensible to seriously consider. It broke an unwritten contract between the company and the user's expectation of how websites worked. You expect to be able to do things like right-click a link and copy the authentic URL, or hover to see where it wants to take you. The notion of obfuscating the link beyond easy recognition and polluting it with tracking markers felt misleading and, well, evil. A move that would mainly only benefit Google, and not it's users. I (quite mistakenly) presumed this opinion would be obvious and self-evident to anyone who spent enough time around the early web to understand its norms.
I explored other ways of achieving the goal, but it clearly wasn't the answer the interviewer sought.
I'm more seasoned now, and experienced enough to say with confidence the approach was wrong. This may seem like a small thing, but a series of misteps and chronic failure to adequately advocate for users is what has led us to the toxic waste dump that so much of the Internet has become today.
I'm really glad to have fresh alternatives (like Kagi), and can't wait for the cultural zeitgeist among developers to swing back around to valuing users as human beings and living up to the trust they place in us.
I was suspicious when they started obfuscating URLs in their own browser, then on their SERPs, and now this...
For many years, I had my filtering proxy rewrite the URLs in the way mentioned in the article.
Almost exactly a year ago, Google stopped working without JS. I stopped using Google.
Now they're upping the game, and as the article (which is a bit of marketing itself) admits, those who have the resources can still blast through these obstacles while those who don't are locked out.
Since the article brings up "AI scrapers", I'll just point it out as being the latest scare-tactic for coercing people to give up the privacy, anonymity, and (browser) freedom of an open interoperable Internet.
Direct URLs in Google search results have been replaced with redirect URLs in the form of www.google.com/goto?url=<opaque base64 string>.
The base64 data appears to consist of a very basic protobuf structure, containing a long string of bytes in field 2 which presumably identify the URL.
Sometimes, these redirect URLs take a perceivable amount of time to load, which is very irritating.
While a lot of people are concerned with local model performance, I wonder how feasible is it now to run a local indexed web search? Surely running an old school Google is possible with the beefy AI rigs today. I know the problem will be crawling which would be bottlenecked by the ISP but I use Google to search SO, Wikipedia, programming language docs, Github issues, and AWS docs. I think a feasible workflow would be to build a set of sites of most interest to you and then prioritize those in crawling.
While typing this out I remembered https://en.wikipedia.org/wiki/Google_Search_Appliance which I never personally used but shows feasibility for the idea. I'm pretty sure one of the newly-announced Macbooks is more than up to the task of matching GSA's offering.
I've been using DuckDuckGo for years now, ever since it became noticeable that two different people searching for the same search term would get two different results back from Google. Meaning they were no longer completely reliable: they might show one person a result that they hide from the other person by burying it on page 3 where few people ever look.
DDG's search results have been poorer recently than they used to — I often see completely unrelated results (to the point of my saying "Why in the world did that come back as a search result??!?") starting from page 2. And yet, I still use them, simply because they aren't Google.
So why are we angry about that ? I mean the end result for the users are exactly the same, it matters only for bots.
Google have such a (justified) bad reputation that whatever they do, people assume it’s entishification. I don’t believed it is on that matter.
When Google stopped paid API search a few months ago, I looked for an alternative for my agents that I felt would be sustainable (one-time setup, then out of my mind). I quickly excluded SERP as I feard Google would pull exactly this type of shenanigans to cut them off.
I somehow found Mojeek and settled on it. I had never heard of them. Unlike Kagi, their business model is ads (so they hold no particular moral high ground). But they have a cheap, working paid API.
What I was astonished by is the quality of the results. For my uses, it's undistinguishable from Google. The conventional wisdom is that web search is a Google-sized problem. How did those obscure Brits pull it off?
What are you going to steal from Google?
The Internet has been dead for years and, after the Scrapocalypse, the small living remnants are behind a login wall.
Google can not provide you anything you couldn't find on either your local ZIM archive or the Media you consume.
At this point, we can all just drop SEO [0]. We are writing content for a robot that hides the source of information.
It's sad that instead of searching things other people put up, we're basically asking sam or dario oracle to tell us the truth. The people should be furious. But we've internalized this idea that they are somehow better.
Dont worry in a year or two, Google wont even redirect you to the actual true url, instead everything will be a page with all links rewritten so all http is tunnelled thru them.
Brave search (free) or Kagi (paid) are able to replace Google and not feel like I'm missing out.
Brave has its own independent index which is cool.
Can someone explain why this matters? Not being flippant I just don’t understand why this would be important.
Of course, this move is user hostile because you don't see what you're being sent to.
It's funny. Recently I looked into what Google is doing (see at https://blog.miloslavhomer.cz/how-google-sees-your-site/).
It's a lot of work to get the data to build an index. Why would they give it to everyone for free?
At least they seem still provide results for my searxng instance. I mean sure, they are horrible but duckduckgo just blocks most queries (and I'm the only person using the ip / seraxng instance)...
Next i'll do is to implement tavilly, exa, tinyfish etc. as search engines for searxng. No agents, no mcp, just their search api endpoint.
If I hadn't already ditched Google long ago, this would be more than enough to chase me away.
> Combined with earlier moves like removing &num=100
Removing this made google search horrible to use. I often use command+f to quickly identify relevant search results, but doing it on 10 results at a time is so laborious that I just don't bother using Google search, resulting in less searches and use of other tools instead.
I really like the idea of turning the search index into a public utility. It is one of those natural monopoly coordination problem things. Just quasi nationalise it for economic efficiency. Ofc google can still sell adds against their own ui (like everyone else). Hopefully this move will move that idea closer to reality.
This is such an incredibly annoying and deliberate defect. I want an extension that will let me resolve the true url without visiting the site, or even rewrites the entire page to show the url.
I... Don't see it? It's the result page right? I just search some random string on Google and the results are all direct URLs. Do they get resolved via javascript after page load and replaced automatically? Or am I looking at something else?
Is it a move to sell more of the paid Google Search API calls? If so, that's a sign of distress.
Surely, the cost of serving search results to bots isn't that high
I use ChatGPT for almost all my searches now. I’m not joking.
This seems good. I'm not sure why I should be upset that Google is preventing abuse of its service.
If you don’t like this, just don’t use Google folks. There are alternatives, use them.
Hopefully this also helps against the rampant CTR manipulation that’s made some search results purely a measure of spend.
The udm=web parameter has been doing the job for a while now. It's a shame you have to know about an undocumented flag to get the old behavior.
Been using Brave Search now for a while including its Brave AI and aside of sporadic times I never needed Google (albeit Brave Search is slower to Google, you get used to it)
Google wants to build up a private web here. We already know this from AMP before.
Also, the search results are total garbage now. Just give it a try and you see how useless the UI is, with AI results first, then tons of commercial go-to entries and only then a few links that are often also totally useless and irrelevant. Google optimised towards crap.
We really need to get rid of Google. Qwant results are now a bit better than before, but still not that great, and I hate that its default UI is a clone of Google. We need more alternatives. Let's get rid of Google once and for all - it has disappointed too many people now.
Bing has done that for years. Hated it.
Most people i see using ChatGPT for any search related work in everyday life.
Maybe Google is observing this?
The enshittification continues.
Nothing good ever comes from businesses desperately trying to protect their moats rather than making their products better so they don't need to.
Why can't agents just click on the link and follow it? I don't get it
ISPs can presumably correlate the Google query string with the request following the response to the goto and so make a search index? I guess they would charge too much.
Do any large ISPs use visit data to feed into a search index?
Append (for logged out users) to the title.
I use Google maybe 10 times a month.
Gemini does something similar
> The real URL is in the Location header on /goto. Request that URL. Do not follow the redirect.
what? the Location header is the redirect, no?
I'm confused. Google has done this for several years already. I have a Firefox extension installed that reverts it - it's several years old.
Note: Article published by Autom.dev which, from a quick read of their homepage, seems like it scrapes Google search results in violation of Google's terms of service and sells those results to customers via an API. That's just my quick read of it, though, so this could be wrong.
wow thanks for sharing
No longer ?
Google has been encoding the target url for years.
My monthly reminder to use Kagi instead
I genuinely forget I’m not using Google until I come across articles like this
"google.com/goto considered harmful"
Who the hell uses Google still? Kagi is the way to go! Even though it’s paid it’s worth it.
As much as I am sad that Google died like 15 years ago, I am past the mourning phase. That was when they announced they were shifting from returning websites to "returning answers" and it has been a long slide into shittification
I do enjoy using their free AI. For actual web search I actually like using Yandex. It reminds me of old Google, returning reasonable results and much less "shaping results to please our corpo-political masters". It is surprising to see how much they have stripped from our view - long tail results, actual results for product reviews and not ad spam, no preference for 20 page recipe sites.
There are still illegal streaming sports and movie sites everywhere (who knew) and all other seedy corners of the internet that have been neatly erased by Google. It makes me nostalgic for that brief window of time when the web was truly uncontrolled, when page rank had meaning and you didn't know if your search would return 0 results or 4,000 pages, which you could actually browse.