alt Hacker NewsThe biggest challenge an agent will face with tasks like these is the diminishing quality in relation to the size of the input, specifically I find input of above say 10k tokens dramatically reduced quality of generated output.
This specific case worked well, I suspect, since LLMs have a LOT of previous knowledge with HTML, and saw multiple impl and parsing of HTML in the training.
Thus I suspect that in real world attempts of similar projects and any non well domain will fail miserably.
In my experience it is closer to 25k, but that’s a minor point. What task do you need to do that requires more than that many tokens?
No, seriously. If you break your task into bite sized chunks, do you really need more than that at a time? I rarely do.