The annoying thing I've found with Linux under memory stress (and still haven't found a nice way to solve) is I want it to always always always kill firefox first. Instead it tends to either kill nothing (causing the system to hang) or kill some vital service.
I'm not sure that I'd want the OS to kill my browser while I'm working within it.
Of course the browser is the largest process in my system, so when I notice that memory is running low I restart it and I gain some 15 GB.
Basically I am the memory manager of my system and I've been able to run my 32 GB Linux laptop with no swap since 2014. I read that a system with no swap is suboptimal but the only tradeoff I notice is that manual OOM vs less writes on my SSD. I'm happy with it.
You can bump /proc/$firefox_pid/oom_score_adj to make it likely target. The easiest way is to make wrapper script that bumps the score and then starts firefox. All children will inherit the score.
If using systemd-oomd, you can launch Firefox into it's own cgroup / systemd.scope, that has memory pressure control settings set to not kill it. ManagedOOMPreference=avoid.
https://www.freedesktop.org/software/systemd/man/latest/syst...
There's a variety of oom daemons. bustd is very lightweight & new. earlyoom has been around a long time, and has an --avoid flag. https://github.com/rfjakob/earlyoom?tab=readme-ov-file#prefe...
Your concerns are very addressable.
Linux being... Linux, it's not easy to use, but it can do what you want.
1. Use `choom` to give your Firefox PIDs a score of +1000, so they always get reaped first
2. Use systemd to create a Control Group to limit firefox and reap it first (https://dev.to/msugakov/taking-firefox-memory-usage-under-co...)
3. Enable vm.oom_kill_allocating_task to kill the task that asked for too much memory
4. Nuclear option: change how all overcommiting works (https://www.kernel.org/doc/html/v5.1/vm/overcommit-accountin...)