Android has a proprietary HAL on both ends; the userspace interfaces to drivers are not the same as on "normal" Linux for almost any devices (baseband, video acceleration although this is at least EGL based, cameras, sensors, audio DSP, hardware media codecs, etc.), so by the time you replace every component of the OS, you've just made AOSP without the useful security parts.

> The advantage being, we can manage packages using a regular Linux distro

It's much easier, more plausible, and more logical to sandbox a regular Linux package-managed distro inside of Android, which numerous solutions exist for.