People used to Google for information; before that, they used to read books.
If you use LLMs for that kind of information-gathering purpose, and then do the work to read and check the references, it wouldn't be any more harmful.
The problem is that LLMs respond personably, with a bunch of ego-glaze added on top, and people are too intellectually lazy to dig in any further.
If we changed LLMs so that they were much more neutral like the starship computer on ST:TNG we'd probably see less intellectually harmful behavior caused by their use, but the companies behind them want to sell ego-glazing because most people lap it up like capnip.