The real contention is two fold: investors are collectively pricing out non-investors that would presumably actually live in the home they purchased, and among investors, institutional investors are both positioned to ignore market pressures and constitute too large a portion of the market, restricting even rental availability and affordability of the homes.

The reason institutional investors are blamed is that they usually have significant holdings elsewhere, and demonstrably will just wait out a market, taking loses they write off against their other businesses in the meantime, rather than actually participating in it. That ability to wait out the market on a finite resource rather than participating is usually otherwise only seen during antitrust market activities. The fact that many cities are considering adding significant vacancy taxes on properties, citing specifically this behavior as a driving factor, is pretty damning.

Additionally, it's not necessary for these institutional investors that are ignoring the market pressures to own a majority, or even that large of a share. Any portion they own and ignore the market for is effectively just removed from the market entirely and reduces the remaining pool of what's available. And the second-order effect is an overall damping of market responsiveness since they are simply refusing to respond to the demand half of supply and demand. I don't have the models to run the math myself, but it should theoretically be possible to calculate how much their involvement without participation impacts the prices as a relation to market share, and I'll bet it's a nonlinear result with a steep curve at even low volumes.