>We've seen this before with IdentityServer

Doesn't really matter. For big, distributed apps at work I use Keycloak or something similar, maybe an own authorization service built on OPAL. For small apps I either use an authentication and authorization library I built myself or, if I don't need something too fancy I use Identity (the one MS provides).