Unsure if it's LLMs that fail at tabular data or its just that tree boosting are spectacular at that task.
Non-LLM transformers beat tree boosting - TabPFN, TabFM
https://research.google/blog/introducing-tabfm-a-zero-shot-f...
[dead]
Non-LLM transformers beat tree boosting - TabPFN, TabFM
https://research.google/blog/introducing-tabfm-a-zero-shot-f...