We have this same point made every release. I wonder if models actually are getting better or if these companies are just secretly hamstringing the previous models before releasing a similarly performing model
Try using a model from a year ago.
I think it takes a while to start finding the weak points in a model. It seems fantastic until you use it enough to find out that it hallucinates APIs, or can’t count letters, or uses too many em-dashes, etc. you don’t really know someone (or something) until you’ve lived with them (or it).
Models are actually getting better.