Quantum computers were proposed primarily as a means for physicists to do simulations of quantum mechanics. I don't think the 'they'll be faster at everything' hype came from them, in fact this basically seems to have come from people misunderstanding the point of them. So I don't think physicists have anything to answer for on this, only perhaps pop-sci writers (most of whom don't think have claimed such a thing either).