I stopped using Oh My Zsh for exactly one reason too: It was so freaking slow it had a noticeable effect on my shell and my mood. It frustrated me that I had to stop and wait every time I opened a tab.
So I removed it, then continued using Zsh and whenever I missed something from before I looked up what it was and installed the plugin easily with Homebrew. The whole process took under an hour. I realised I only needed two or three plugins.
Now my shell is fast, without unnecessary bloat, and does what I need. I’m much more productive and happier, and at the end of the day I don’t really see what’s more important than that (within the scope of the conversation).