alt Hacker News30papers.com – Ilya's 30 essential ML papers, in a beginner friendly format
Comments
Hey guys, I really appreciate all of the attention this post has received. I honestly thought it was going to be just a small project to help some of my friends get into reading research papers.
A large number of people complained about how intense some of the backgrounds/animations were (I might have been a bit too focused on making something that looked cool over usability). In response I have added toggles for both the movement on the page and the backgrounds for the papers.
Other people mentioned that they would have liked some more personalised reflections on each paper. I currently have already done some of these for the more popular papers on my X @notmcrowley . I would have no problem adding these to the site if people think it will help. I feel the need to warn that I have not been formally educated on ML or AI so any interpretation will just be mine and may not necessarily be the correct one. (If anyone with more experience would like to contribute to this feel free to reach out).
For beginners I'd recommend the Welch Labs Illustrated Guide To AI if your not well versed in reading papers. Its a beautiful book that I've enjoyed going through. I'd recommend going through these papers after reading that to get a deep understanding.
Author here. First year CS student at Trinity College Dublin. I Built this because when I was getting into reading research papers I ended up burning a ton of my Claude usage asking questions other people have probably already asked. The website is just a side project and definitely a WIP. Happy to answer questions or take PRs on GitHub.
I wish this were organized according to suggested/logical reading order. For example, the paper introducing the attention mechanism probably ought to precede "attention is all you need".
I was confused for a minute, I thought this was "top 30 papers by Ilya" and was then wondering why "Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton" is on the list.
> In additition, even though I have read the vast majority of the papers featured on the website, I have not read through each of the website's versions end to end.
Website's versions, as in - the actual text or the "explanations"? Either way this is a big red flag.
After seeing this for the first time, I've build PdfToMp3 to listen to these papers. It has now evolved into ListenDock. Fun fact: PdfToMp3 existed before NotebookLM and I already had "overviews", but I called them teacher explanations.
Here is an example of a "Teacher Explanation" of the paper "Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton"
https://listendock.com/e/quantifying_the_rise_and_fall_of_co...
Noting the theory papers on Kolmorogov complexity. For those not familiar, Ilya argues that the reason why neural networks generalize -- why they work at all -- is because they are effectively finding a simple description of their training data, converging down onto the limit of the Kolmorogov complexity. [1]
Nice presentation of the list!
I'd recommend watching a few of his talks/podcasts before during reading these to get the overview and how all the bits in these works tie together.
https://www.dwarkesh.com/p/ilya-sutskever
https://simons.berkeley.edu/talks/ilya-sutskever-openai-2023...
So the styling and animation work looks really cool (when isolated), but they distract from the content itself, IMO.
I think it'd work better if you featured the animated background effect toward the top of the page and shifted toward static graphics (or much subtler animations) as the user scrolls.
And I don't think the zoom-out effect on the listing cards has the intended effect; I found myself wanting to get a better look at the papers and was a little disappointed/annoyed when they got smaller and harder to see as I pulled them into view.
The colors/shadows/layout all looks really nice, but I feel like the animations (as-is) ultimately detract from the experience rather than add to it. Thanks for sharing, though!
I thought the actual 30 papers have never been disclosed. Do you have a source tying the recommendations back to Ilya, or did you come up with this list?
No need stupid moving texts.
CS231n: Convolutional Neural Networks for Visual Recognition - https://cs231n.github.io/
The Unreasonable Effectiveness of Recurrent Neural Networks - https://karpathy.github.io/2015/05/21/rnn-effectiveness/
Understanding LSTM Networks - https://colah.github.io/posts/2015-08-Understanding-LSTMs/
ImageNet Classification with Deep Convolutional Neural Networks - https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436...
Deep Residual Learning for Image Recognition - https://arxiv.org/abs/1512.03385
Multi-Scale Context Aggregation by Dilated Convolutions - https://arxiv.org/abs/1511.07122
Identity Mappings in Deep Residual Networks - https://arxiv.org/abs/1603.05027
Recurrent Neural Network Regularization - https://arxiv.org/abs/1409.2329
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin - https://arxiv.org/abs/1512.02595
Order Matters: Sequence to Sequence for Sets - https://arxiv.org/abs/1511.06391
Neural Machine Translation by Jointly Learning to Align and Translate - https://arxiv.org/abs/1409.0473
Pointer Networks - https://arxiv.org/abs/1506.03134
Attention Is All You Need - https://arxiv.org/abs/1706.03762
The Annotated Transformer - https://nlp.seas.harvard.edu/annotated-transformer/
Neural Turing Machines - https://arxiv.org/abs/1410.5401
A Simple Neural Network Module for Relational Reasoning - https://arxiv.org/abs/1706.01427
Relational Recurrent Neural Networks - https://arxiv.org/abs/1806.01822
Neural Message Passing for Quantum Chemistry - https://arxiv.org/abs/1704.01212
Scaling Laws for Neural Language Models - https://arxiv.org/abs/2001.08361
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism - https://arxiv.org/abs/1811.06965
Keeping Neural Networks Simple by Minimizing the Description Length of the Weights - https://www.cs.toronto.edu/~hinton/absps/colt93.pdf
A Tutorial Introduction to the Minimum Description Length Principle - https://arxiv.org/abs/math/0406077
The First Law of Complexodynamics - https://scottaaronson.blog/?p=762
Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton - https://arxiv.org/abs/1405.6903
Kolmogorov Complexity - https://onlinelibrary.wiley.com/doi/book/10.1002/047174882X
Variational Lossy Autoencoder - https://arxiv.org/abs/1611.02731
Machine Super Intelligence - https://www.vetta.org/documents/Machine_Super_Intelligence.p...
Its interesting seeing how many of these researchers became the heads of frontier labs!
Kolmogorov Complexity looks interesting. It seems to formalize Occam’s Razor and the notion that intelligence = compression.
Is this meant to be read in order?
Anyone got a list for the agentic LLM age?
Where did you get the list? AFAIK, list was never shared
The formatting of the articles on this website is bad. I've opened the first one and all the LaTeX formulas are messed up. The subscripts and superscripts are all flattened rendering the math hard to comprehend. Did the author actually try to read any of the articles?
>∏ plocal(x|z) = i p(xi|z,xWindowAround(i))
Images and tables are not rendered at all. What is the point of this? Just keep the links to arxiv and leave it at that, otherwise render the articles properly
Why on earth would you deliberately choose to do whatever the fuck it is you did with the scroll and the animations for each paper when scrolling through the landing page? What are those animations supposed to be? I use firefox but I also visited on chrome, and the page is even more broken there. Scroll doesn't "take" unless I scroll hard enough, otherwise it bounces back. But on chrome, at least, it seems like the animation for each paper is clearer - it's supposed to be animating the scale of the paper as you scroll to it.. but it seems that your background animation is lagging everything so much it just doesn't work.
Main page UX is terrible. If you go for quirky, fine, but I would not want to use it.
[dead]
> "beginner friendly format" > looks inside > math
Someone posts on X, "These are Ilya’s 30 papers", gives no source, doesn't say where he got it from, and isn't connected to either Ilya or Carmack (Ilya gave him the list).
Then someone vibe codes a barely usable website based on that, and it lands on the HN front page? Is this correct?