Going through dating, marriage, having children, growing a family, co-parenting children is a massive time and emotional investment (for most people who take it seriously). A large part of, at least Western, society is built on this concept of spousal trust. It's part of the tax code, its part of the legal code (spousal privilege), it's part of our moral fabric, it's a cornerstone of many religions. Breaking that is a huge betrayal of our collective societal moral understanding.