Well, technically if you only have a mutable borrow (it's not your object) then you can't move from it unless you replace it somehow. If you have two such borrows you can swap them, if the type implements Default you can take from one borrow and this replaces it with its default and if you've some other way to make one you can replace the one you've got a reference to with that one, but if you can't make a new one and don't have one to replace it with, then too bad, no moving the one you've got a reference to.
You're right and I edited my comment.