alt Hacker NewsWin32 has managed to do this without any API change, all the existing APIs work. The same approach would've worked for X11.
What it does is simple - all the functions that deal with windows/handles or events simply do not work on ones that you don't have access to, for example, the EnumWindows function allows you to wall through the tree of windows simply do not see the ones the process has no access to. SetWindowsHookEx which allows you to intercept and modify messages meant for other windows simply doesnt fire for messages you're not supposed to access.
Granted, outside of UWP apps, the application of security is rather lax (this is for legacy purposes, the security's there, just not enforced), but for apps running as admin, or UWP apps, the sandboxing is rather solid.
Indeed, this is the right approach.
Moreover, it is possible to choose as the default policy that no program may access a window that it did not open, but then there must exist a very simple method for the user to specify when access is permitted, e.g. by clicking a set of windows to grant access to them.