For that to be meaningful you'd need evidence directly showing knowledge of the affair prior to the public leaks. If you had evidence of cover ups that would also make it a much stronger case compared to having known or heard rumors and done nothing.
The board said he mislead them for over a year on the affair(s). That's more than enough to be meaningful for a company board who wants to avoid the liability he was in the process of bringing on them.