I think it matters to more than you might think. A significant portion of the non-technical ChatGPT userbase get really attached to the model flavor.

The GPT-4o controversy is a good example. People got attached to 4o's emotional and enthusiastic response style. When GPT-5--which was much more terse and practical--rolled out, people got really upset because they were treating ChatGPT as a confident and friend, and were upset when it's personality changed.

In my experience, Gemini and Claude are much more helpful and terse than ChatGPT with less conversational padding. I can imagine that the people who value that conversational padding would have a similar reaction to Gemini or Claude as they did to GPT-5.