> what Windows actually expects the vtable to look like
I'm not a Windows-knower, are vtable layouts part of the user<->kernel ABI?
It’s part of the MSVC x64 ABI, not a part of the kernel ABI. The kernel and user mode (Win32) APIs all use C linkage so vtables are not relevant and the ABI is a convention of the compiler not the OS. Practically speaking though if your software runs on Windows it will probably use the MSVC ABI.
Yes, you either follow Itanium ABI (non Win) or MSVC ABI. Technically nothing stops _your_ compiler from implementing totally own conventions but it will only be compatible with itself
All of the major kernel ABIs are using C function interfaces as their stable ABI (except Linux, which uses an assembly syscall instruction as the stable interface, although you still rely on a lot of the ancillary C ABI for things like struct layout or stack layout).
For C++ ABIs, there are really only 2.5 major ABIs: the MSVC ABI, used by MSVC and clang wanting to be compatible with MSVC, and the Itanium ABI, used for everything else. There are some slight variants on the Itanium ABI which makes the ".5": ARM uses a different layout for exception handling tables, and there are some flags you can set to use a more compressed vtable layout (32-bit offsets instead of 64-bit pointers). (There are other older ABIs, but either companies stopped making a C++ compiler or they switched to Itanium.)
Windows COM APIs (which isn't part of the kernel, they're still userspace libraries) rely on an IDL which is meant to be directly compatible with the C++ vtable. That said, they also use a restricted subset of C++ that all of the ABIs are going to agree on for vtable layout--if you don't overload any functions, and you don't have any virtual inheritance, there's pretty much only one possible sane vtable layout, and everyone does that.