They likely won’t have a full pretrain for awhile. Just like everyone else, the name of the game now is to finetune existing models.