Firefox is set to allocate memory until a certain absolute limit or memory pressure is reached. It will eat memory whether you have 4GB of RAM of 40GB.
Set this to something you find reasonable: `browser.low_commit_space_threshold_percent`
And make sure tab unloading is enabled.
Also, you can achieve the same thing with cgroups by giving Firefox a slice of memory it can grow into.