Your sense of worth should not be tied up in your job.
“I wish I had spent more time on my business” is totally different from “my work gives me a sense of purpose and makes me feel like I’m contributing.”
It's not like it's what you spend most of your life's hours doing or anything like that.
What I would say, and perhaps what you actually mean, or if not then at least what you might now consider, is you shouldn't give anyone else the right to declare your sense of worth, including an employer.
It’s just an excerpt and not applicable to some but Rainer Maria Rilke’s Letters to a Young Poet has a good if longer bit: “Are not all careers the same? Filled with demands and people filled with animosity towards the individual. At the same time absorbing the hatred of those who have silently and sullenly adapted to dull duty.”