he wouldn't have publicly apologized if all of that was "perfectly fine" with his wife