I empathize with that situation, it's something that weighs on my mind a lot as well. One of my goals is definitely to get the mortgage cleared out sooner rather than later. In some ways I'm rolling the dice, but I also feel like I'm safe from layoffs at least one quarter at a time (e.g. I don't think it will be so sudden, it'll align with quarterly earnings), and anything I can pay off in a quarter of work is worth rolling the dice on, even though there's always some risk there.

My plan of the moment is to get to a point where my expenses are minimized (e.g. house is paid off) and I can cover basic living necessities on investment income, and do some type of artisan home business. I keep thinking I'm going to really seriously get into carpentry, which has been a long-time hobby, or open a mechanic's shop (my other long-time hobby).