I don’t know why this wasn’t mentioned before but why not use a Firewall. If you’re using a virtual box like Hetzner or Scaleway you can specify an ip or range at the router level. For all intents and purposes this removes public exposure. Scaleway also has a cheap VPN bridge. So you never need to connect via the public internet if you don’t want to… hardly gets more secure than that