... so the real problem is that to get back and slow down nicely would require so much more launch mass, right?

(By slow down I mean to change to an orbit that has more drag and wouldn't take forever to return to Earth.)