The killer with exercise is that it makes you hungry, proportionally to how hard you worked out.
It's a real killer for "calorie blind" people to go on a 750 calorie hike, and then stop by for a "treat" on the way home. That treat being a 500 calorie blueberry muffin with 200 calories of butter on it. Then they go home and have dinner.
Blame evolution. For 99% of our 300,000 year history, starvation not obesity was what killed homo-sapiens.