alt Hacker NewsPageRank explained
Comments
I actually built this, and shipped it, in 1996, with no knowledge of page rank, citation analysis, or bibliometrics, for an internal/external search engine for the Envirolink web site. Envirolink was a directory of environmental web sites, so they already had a list of URLs to crawl. The reason it was feasible to build was that it was a fairly constrained list of URLs, it wasn't the entire web.
I didn't really know what I was doing (I was 17), but it was an awesome unpaid summer internship. There were two parts of the search engine - a crawler and the search engine. Both were written in Perl.
PageRank is fascinating, since it is so easy to explain.
Yet, this is not even half the work. It like a third of the way.
Before you could have invented PageRank, you must think in graphs. That is possible in 1996, but not as widespread as today.
After you invented PageRank, you still need to deploy it. Again, possible but challenging as well. Is Python performant enough in 96? Can you afford more than 4MB RAM?
At least Lego will not sue you for using their bricks to build a server rack in 1996.
Here are two excellent videos that explain and visualize the PageRank algorithm:
* [2020-06-17] Spanning Tree - "How Google's PageRank Algorithm Works" (5m16s): https://www.youtube.com/watch?v=meonLcN7LD4
* [2022-05-23] Reducible - "PageRank: A Trillion Dollar Algorithm" (25m25s): https://www.youtube.com/watch?v=JGQe4kiPnrU
Well, I was a child in 1996, so probably not.
Tying relevancy to link frequency was definitely a novel idea at the time, even if it seems "obvious" or simple in retrospect.
As, I think, Page points out in the patent, PageRank's idea comes from Science Citation Index. That was an inverted list of scientific references, where you could look up an scientific paper in an expensive set of bound books and find all the papers in which it was later referenced. You can then use this to see who's getting referenced a lot, which is an ego trip in academia. Academic libraries had copies of that index. Now everybody has that kind of info, but when it had to be done by hand, it was hard.
Inverting the huge, sparse matrix of references for PageRank was expensive. Originally, Google did it about once a week. The big breakthrough was when someone (who?) figured out how to do it incrementally at scale.
Thinking of an algorithm in the abstract is one thing, implementing it at scale is another.
Yes you could have invented PageRank, but could you also have invented MapReduce, BigFiles/Google File System (GFS), Google Web Server, Bigtable, Protobuf? Then spun up fault-tolerant clusters consisting of cheap commodity PC hardware in an era where AWS wasn't even an idea yet? Then invented the concept of Borg to manage this hardware globally?
Also PageRank as a Markov chain
[0] https://math.libretexts.org/Bookshelves/Linear_Algebra/Under...
It's giving "Attention is all you need" but in the year 1996
Related, fascinating foundational work
Citation index https://en.wikipedia.org/wiki/Citation_index
Shepard's Citations https://en.wikipedia.org/wiki/Shepard's_Citations
Yes anybody can be in the right place at the right time.
You could also have invented a wheel because it's so obvious.
Let's not forget that PageRank unintentionally spawned the SEO industry.
What I always wonder is, how does google knows how to go to abc.com, xyz.com, mypage.com, etc. so it can crawl them?
> Sergey Brin and Larry Page came up with this precise algorithm, i.e., PageRank, which was one of the key algorithms that helped catapult Google into a household name and made them tons of money. Both Sergey and Larry were grad students at Stanford, so their coming up with such an amazing algorithm doesn’t seem surprising.
No, Brin wasn’t a co-inventor of PageRank.
One is reminded of Damien Hirst's famed retort to a critic who said "Well I could have pickled a shark" ... "But you didn't, did you. I did."
I could not have invented it. I was one year old.
How do you compute page rank for billion of pages that have cyclic links? Is that not the problem right after the initial idea?
Yup. I was there, using the web, back then. I used Altavista dozens of times a day. It wasn't very good.
I never thought of it.
I never thought of the Million Dollar Homepage, either.
It's still there! https://milliondollarhomepage.com/
I remember the first ever HTML CV (résumé) being published.
I am Slashdot user #6030. I used it for ages before I created a user account.
I was already paying for my own personal email address in 1991 when timbl revealed the WWW to the world. I thought it was a gimmick. It'd never catch on. We already had Gopher and Archie and Veronica.
deep sigh
No mention of RankDex?
curious, how do you decide on which page to start from? Popular directories?
All credit to you for writing this yourself!
PageRank, at its core, symbolizes these basic properties.
Step 1: be sponsored by the CIA
it’ll give you an article on “Hotels for Chickens” if you search “Hotels” because the word matches
What if I fricken want fricken "Hotels for Chickens", what about that, huh?
I mean, page rank was useful but it was also a step in the direction of "search give you what it think you, not what you asked for" and I think now we can how far and dubious progress in that direction has been.
[dead]
If I had received funding from DARPA, and NASA, perhaps I could have.
Importantly, PageRank doesn't work today - you need something else. It was one of many possible ranking hacks, and one that worked at the particular time in that particular state of the web where nobody was gaming links because PageRank didn't exist yet. Maybe you could invent a good ranking algorithm for the modern internet, perhaps just the reciprocal of the number of ads on the page, minus its AI detector score, but probably not that.
Unimportantly, it's named after Larry Page, not after the fact that it ranks pages.