Set `browser.low_commit_space_threshold_mb` and/or `browser.low_commit_space_threshold_percent` to something you'd prefer, and confirm that `browser.tabs.unloadOnLowMemory` is set (I think it is by default).

The default settings are to allow it to acquire memory until memory pressure on the system reaches 5% free, at which point it will begin freeing memory. You can set a custom percentage or a specific amount of memory.

That or just run it in a cgroup with a memory limit.