If a judge says you're in contempt, you'll get charged with contempt immediately - all the people required are present.

To charge him with defrauding investors requires a whole different group of people to get involved.

Additionally, those people need enough evidence to have a chance of conviction. "He refused to answer questions about it" is not actually evidence.