> how is running servers supposed to be 0 cost, while running ai inferrence isn't?

For a SaaS business, running servers isn't free. But compared to the cost of running GPUs for inference that you are selling, it almost is. The company I work for is a SaaS company. We have a single production server. A couple of QA servers. All hosted on Hetzner. Monthly cost for servers is less than $400. This generates a few million dollars a year in revenue.

If we were in the business of selling inference, our cost of providing the service, for the same amount of revenue would significantly higher.

Even large businesses like Microsoft, Meta, Google have operated with similar margins. Cost of running servers, compared to revenue was very low. But inference changed that, in a dramatic way.