The city of London literally has a city also named London inside it and yeah, "Where are you going?" "The City" means clearly that tiny part. There is another small older city inside London, Westminster, but if you meant Westminster you'd say "Westminster" never "The City" in general.

I grew up equidistant from Philadelphia and New York. "The city" was exclusively used for New York regardless of the many cities between us