earlyoom is what you're looking for. it'll crash the process instead of hang