alt Hacker NewsOver at https://relflow.ai (an OSS model factory) used Kapso to help harness an HPO loop on RelArena. RelArena is like RelBench, but is maintained and regulated by Prior Labs to isolate data leakage that can taint modern benchmarks.
Kapso was able to efficiently calibrate Relflow's hyperparameters on 21 structured data modeling tasks, and we were able to outperform TabPFN on the majority of the larger data modeling tasks despite not using any ICL, and using ~1% of the number of params of TabPFN.
Thanks for your comment. We have also tried it ourselves in relbench (Stanford's predictive modelling on relational databases), and surpassed tabular foundational models: https://github.com/Leeroo-AI/kapso/tree/main/benchmarks/relb...
Our hypothesis is that similar to RL principle, you can experiment in the environment (predictive modelling on relational databases), learn from it, and compound your domain harness with these bank of lessons, instead of re-training foundational table model, which puts these learnings in model weights. This enables faster iteration, and transparent learning over time.
As a proof for this domain, Kapso leads in classification, regression and recommendation tasks vs. best tabular foundational model (Kumo AI/Nvidia): https://star-project.stanford.edu/relbench/leaderboard/#clas...