You're sort-of right, I think, because you do need an Apple account to sign in to the Mac App Store to get current Xcode in the first place - but the $99 is entirely optional!

For distributing your program without the fee, you'll probably moan about the hoops that people have to jump through to run your stuff: https://support.apple.com/en-gb/guide/mac-help/mh40616/mac - and I can't say I love this myself, but people can run your stuff, and no fee necessary.

(I've got a couple of (somewhat niche) FOSS things for macOS, and I build the releases using GitHub Actions with whatever default stuff the thing uses, then make up DMGs that people can download from the GitHub releases page. I added a bit in the documentation about visiting the security dialog if you're blocked - and that seems to have been sufficient.)