Give it time. If I do another startup, I’m not even going to bother renting an office. For me the answer is pretty obvious, all work that can be remote should be remote.
And the situation you’re describing can also happen on-site. I’m in that situation now, where I (a very senior IC, by choice) am expected to both deliver the “IC” stuff, and supervise 3 other people because their manager can’t tell a softmax from a hole in the ground.