I think you as the user can use the aliases without Sign in with Apple though, right?

But otherwise, you're right, any website that wants to accept Sign in with Apple will almost certainly be agreeing to Apple's TOS for Sign in with Apple I presume will stop you from blocking this service.