My experience has been that it's easy to say, "oh, it's just me", but much harder to subject someone you care about to the same standard that you would yourself. I'm in a similar position with the thermostat, even though something we initially bonded over was that we both kept our thermostats at a low temperature that was outside the window of being socially acceptable.
In the winter, I keep the thermostat at frigid temperatures when I am home alone and jack it up to warm just before any one else gets home. My thinking is that it is wasteful to warm up the entire house when it's just me since I can put on a sweater but I don't want to subject others to my, shall we say, quirks.
I keep meaning to calculate how much I am actually saving by freezing my butt off. My guess is it'll work out to something like $0.75 a day or something equally trivial.