I was involved in a very similar situation once. I recommend wireguard for this, it's mature for years, has superb support in linux and some BSDs and there are userspace implementations if you need that. It wraps traffic in UDP, the overhead is much smaller thus throughput mich higher than traditional TCP-based VPN (you want to avoid tcp-in-tcp!). There were once patches posted to lkml that passed QoS-flags from the inner packet to the wireguard packet, if you need that. not sure if that landed upstream in the end. key distribution and lifecycle management is what was still unsolved years back when this was evaluated, nowadays tailscale and its clones and similar oss should serve you well.