I like to re-add the readline binds after enabling vim mode, and then I only leave insert mode occasionally. C-x C-e really is great, though, I think I do tend to reach for that if I need to do a big edit of a one-liner.