Most Swift compilation slowness can be avoided by componentizing using packages, plus a few tricks like being explicit with types rather than relying on type inference.
I’m not sure why you’re having so much trouble with SPM though, most people seem to get on well with it.
Trying to make a .xcframework from SPM is "fun". And getting the Objective-C that are generated, but treated as intermediary artifacts discarded is a bit of a pain.
But I guess the issue here is that .xcframework is an Apple thing, not a Swift thing.
The whole "won’t build if the product / target has the same name as one of its classes" is absolutely ridiculous on the other hand.
Experience with SPM might vary depending on how many dependencies you’ve got. A lot of Apple platform apps are quite thin on third party library use, in which case SPM is probably not a large source of trouble.
SPM is fine for most Swift and then fully enraging when you have a slightly divergent use case. IME of course.
I'm also wondering how big their programs are that the compile times are an issue. I'm seeing fantastic build times with Xcode 16 and Xcode 26.2 with swift 6.2 it's even better. SPM is also better with the newer version. Most issues can be solved or exposed by closing the window and reopening or going to the package with issues and pressing cmd+s.