TLDR: I agree.

A library might also be used in multiple place, maybe deeply in a dependency stack, so the execution context (high level stack) matters more than which library got a failure.

So handling failures should stay in the hands of the developer calling the library and this should be a major constraint for API design.