It's nuanced. I'm not an attorney and I don't have the bandwidth right now to go looking for citations. The general read I'm getting is that zealously defending your rights to the mark are the safest way to make sure a court doesn't see you as abandoning your rights. I gather that different courts in the US have treated a lack of defense differently.