alt Hacker NewsDoes optuna think abstractly (i.e. use LLM to interpret the code and come up with insights), or just perform hyperparameter tuning experiments on user-indicated parameters?
Does optuna think abstractly (i.e. use LLM to interpret the code and come up with insights), or just perform hyperparameter tuning experiments on user-indicated parameters?
The latter, but it uses fairly optimized approaches to ensure it selects the best candidates.
If you look at the commits, you can see that all it does is just set different values for different parameters of continuous values: the type of thing that I trust statistics a lot more than reasoning. Optuna can make very informed decisions when making lots of different changes at once, slowly converging towards optimal parameters, where the LLM seems to be throwing stuff at a wall and see what sticks.
What would work best if the LLM would try to approach things on a higher level, ie use Optuna, but reason about better approaches for algorithms and/or data or whatever. But what it ends up doing is tuning parameters manually, only one / a few at a time, extremely inefficient and unlikely to be optimal.