I just had to rip out a lot of native Swift concurrency, and replace it with classic GCD.
Looks like tasks mess with the reference counter. My app was not releasing memory, and was constantly jetsam-crashing.
Also, the app was quite sluggish.
Reverting to GCD, fixed both these issues.