This is called an oomkiller. The kernel has one but it kicks in very late and the kernel prefers to do page trashing instead of killing processes.
systemd-oomd should be integrated in systemd, you can configure it to your liking and see if it improves your problem.
Adding to this, glad systemd-oomd finally added solid rulesets in 261
Made it far easier to target any containers that got too hot rather than ever risk anything higher priority.
I believe that you mean: https://en.wikipedia.org/wiki/Thrashing_(computer_science)
Chris Siebenmann discusses when the OOM killer triggers: https://utcc.utoronto.ca/~cks/space/blog/linux/OOMKillerWhen
Chris disables systemd-oomd after it obliterates his X session with no explanation: https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdOomdNo...
I wish there was an easy way to configure it to say "target user processes first, specifically java (or these days python)" as in my experience they are always the culprits. Processes owned by system accounts or root should be the last ones killed.