I think what you're describing here is mostly a mindset / user education problem. SRS is for "serious learning", which by necessity will unconditionally feel difficult - if your training sessions aren't strenuous, then they don't drive adaptation.

It's hard to get around without marrying the SRS with something like a hierarchical skill tree whose traversal you can be made aware of, or some other visible progress metric (eg, climbing the ELO of encountered puzzles in a chess training engine).

Still: users have to get comfortable with being uncomfortable if they want to profit from these sorts of systems.

A different issue with SRS's lazer accuracy is the Pareto tradeoff between efficiency and robustness.