A library OS is an OS that is linked directly to your program instead of being a separate program accessed through a syscall to kernel mode. About the same as a “unikernel”, but a more recent term.

Basically it lets your program run directly on a hypervisor VM, though this one will also run as a Linux/Windows/BSD process.