It's a library that is linked to in place of an operating system - so whatever interface the OS provided (syscalls+ioctls, SMC methods, etc.) ends up linked / compiled into the application directly, and the "external interface" of the application becomes something different.

This is how most unikernels work; the "OS" is linked directly into the application's address space and the "external interface" becomes either hardware access or hypercalls.

Wine is also arguably a form of "library OS," for example (although it goes deeper than the most strict definition by also re-implementing a lot of the userland libraries).

So for example with this project, you could take a Linux application's codebase, recompile it linked to LiteBox, and run it on SEV-SNP. Or take an OP-TEE TA, link it to LiteBox, and run it on Linux.

The notable thing here is that it tries to cut the interface in the middle down to an intermediate representation that's supposed to be sandbox-able - ie, instead of auditing and limiting hundreds of POSIX syscalls like you might with a traditional kernel capabilities system, you're supposed to be able to control access to just a few primitives that they're condensed down to in the middle.

I think that's an OS in the form of a library, like Wine for example. From what I get from the description it allows you to run programs on your real OS and make it see a cut down API to your actual system to reduce the attack surface.

https://en.wikipedia.org/wiki/Operating_system#Library