In fact, part of accepting a pardon is accepting guilt. That can particularly be consequential if there is a civil case associated with the criminal charges. For example, if I'm charged with drunk driving and I run into someone's house, by accepting a pardon I have to admit that I'm guilty of drunk driving which the home owner can then use in their civil suit to extra money for the damage I caused.

This is part of the reason why people will sometimes not accept a pardon.