alt Hacker NewsI'm afraid it might be the other way around. RSI might pick all of the low hanging fruit soon. There must be a physical limit of how much intelligence you can squeeze out of some amount of parameters and compute.
There are going to still be worthwhile improvements but they are going to be more like not how to make transformers 10x cheaper but how to make next training run cost 9 trillions instead of 10 with a very particular optimization designed at the cost of hundreds of millions for this one specific run.
Replies
It is just occurring to me that “RSI” expands to recursive self improvement. Thought people were talking about repetitive stress injuries; either in regards to programmers writing too much code/not having to write code anymore, or the frontier AI companies and their tendency to applaud themselves.
[flagged]
I think we’re no where near a physical information theoretic limit.