As far as I understand you cannot use IP address as the outer certificate as per https://www.ietf.org/archive/id/draft-ietf-tls-esni-25.txt

> In verifying the client-facing server certificate, the client MUST interpret the public name as a DNS-based reference identity [RFC6125]. Clients that incorporate DNS names and IP addresses into the same syntax (e.g. Section 7.4 of [RFC3986] and [WHATWG-IPV4]) MUST reject names that would be interpreted as IPv4 addresses.