I don't think anyone ( at least around me ) is drinking milk based drink twice as much just because they feel like they get less energy per drink from skimmed milk.

You are making an argument that people do so, do you have any evidence for this ?