I'm just getting horrible segmenting and far-pointer vibes of the whole thing, been coding a classic Gameboy game for fun so fiddling with memory mappings is part of the "fun" but for anything non-constrained I'd hate that.
We buried far pointers with DOS and Win16 for a good reason..
I'd take segment-pointers over copying megabytes of memory around anyday though ;)
It's not much different than dealing with all the alignment rules that are needed when arranging data for the GPU.
"Far" memory is already a central concept in modern GPU APIs. Not all GPUs naturally share memory with the CPU they're running against and handles for resources are a requirement.