Is the hallucination or “being wrong” the only major problem? Because if that’s the case I can’t imagine that problem will still be around in ~1-5 years.

I think it’s actually very similar to a script. It takes an input and produces an output. Certainly it has more variation than a traditional script, but it's generally going to return a very similar answer when the same input is given repeatedly, like when you ask an employee at a restaurant where the bathroom is. The verbiage may not be the same every time but the answer is always the same.