alt Hacker News> running Pangram against every article that enters The Consensus feed is financially impossible.
Surely reading every article in the feed is even more costly than running Pangram on it? If there are so many articles that Pangram is too expensive, that would also suggest that there are too many articles for them to be meaningfully curated, which seems like a bigger issue.
For sure, there are ~23,000 articles in the feed (which only goes back 5 years, but grows naturally as people publish and as we index more companies in software infrastructure).
The point is to index interesting jobs and articles to 1) help people follow the technologies they're interested in and 2) to help us figure out what to write about.
Articles get tagged with mentioned infrastructure technologies (e.g. Postgres [0] or Lean [1]) and get tagged semantic categories too (e.g. compilers [2] or network protocols [3]). (And all the same tagging happens for jobs as well.) The weekly newsletter [4] features articles and jobs from a few key semantic categories. But you can also build your own RSS feeds or email watchlists for whatever you're interested in.
There's a lot of filtering done to keep out marketing content, listicles, etc. And checking AI tells is the next stage of that.
Limiting to only good quality articles lets us build broader intelligence like the ranking of programming languages [5] and databases [6] and helps us figure out which companies are most focused on the technologies and so on.
[0] https://theconsensus.dev/feed.html?q=projects%3DPostgreSQL
[1] https://theconsensus.dev/feed.html?q=projects%3DLean
[2] https://theconsensus.dev/feed.html?q=topics%3D%22Compilers+%...
[3] https://theconsensus.dev/feed.html?q=topics%3D%22Network+pro...
[4] https://theconsensus.dev/weekly.html
[5] https://theconsensus.dev/languages.html
[6] https://theconsensus.dev/databases.html