The bug is that Google’s own website outside of the context of user generated content becomes the source of the link and that alone removes a large amount of the suspicion.

I think the author of this attack could easily modify it to be way worse.

Just change it to inject a message saying “you have run out of creator studio AI credits, please add on a Geminin Creator Plus plan to continue. You will be taken to a third party billing service to complete the transaction” and then link to a malicious billing page.

I find this apathetic response from Google to be pretty confusing coming from one of the big AI companies making a big stink about AI safety. How about trying practicing what you preach and make your AI safe? Or were those all dog whistles for regulatory capture?