alt Hacker NewsIf you can do it offline, batching solves this. We used a test dataset from CFPB and at n=20, it was 1.6x faster and 1.2x more expensive with no statistically meaningful accuracy dropoff. Did not tune for batch size, but it's possible that we could get to 30 and see better perf.
That's not one if, it's two big ifs. If you can to it offline. If you can batch. I wouldn't say that's all that common in the use cases Jev sells itself for. You often want the answer now, not "averaged across 20 items in a batch that took 20 times longer".
It's fair to note it, but I wouldn't say "batching solves this" so much as "batching can solve time insensitive use cases".