Oh yes, I didn't clarify that I meant an argumentum ad hominem, which isn't the same thing as the informal fallacy of the same name. So yeah, I agree, this isn't directly fallacious.