Keeping the key in the same room as the padlock only protects against casual drive theft and secure disposal.

Personally I'm more worried about someone stealing the entire server or a local threat actor.

Sure, keep TPM to help with boot integrity, maybe even a factor for unlock, but things like Clevis+Tang (or Bitlock Network Unlock for our windows brethren) is essential in my opinion.