alt Hacker NewsSee their revenue (number of paid users is not a secret): something around $7M annually? It was half of that not so long ago (glad, that the userbase is growing).
With their current pricing they are out of their league of having any full-blown index, crawlers, people, what have you.
I would say year ago I was amazed how they are alive at all (unless I am missing something in their funding).
A little over 15 years ago you could index the web with a small cluster. I remember people doing doing it with Cassandra or Elasticsearch. I'm sure you'd need a much bigger cluster, but outside video and images I imagine it's still doable even for a small organization, especially if you're filtering out content farms. Plus, there are many organizations interested in having access to an index, and I'm pretty more than a few currently running their own index and selling to analytics firms.