Such a “poison” could indeed be very powerful. While the models are good at incorporating information, they’re consistently terrible at knowing they’re wrong. If enough bad info finds its way into the model they’ll just start confidently spewing junk.