I'm talking about handling the OAuth token exchange and what not. Your identity server doesn't need to be a separate service, it can be integrated into your backend. At the end of the day it's responsible for making sure client claiming to be X is X. I really don't think this is rocket science.
I've set up email/password, email magic links, SMS 2FA codes, OAuth, and it's never been this magical, mystical thing these "auth providers" want to pretend it is.
Yes, you need to wire up endpoint for you to feed data to the library (tokens), and provide ways to refresh tokens, etc but that's all just wiring up and I don't think that's really that hard (even before LLMs).
I just cannot fathom handing over as much control and third-party auth providers require you to.