This is how LLMs have been since the beginning. They can be really okay at the most surface level information, but as soon as you dig any bit beyond that, it all falls apart. Inquire any chatbot about something that you're an expert in and it becomes quickly apparent.

I'm here for using LLMs to take some of the tedium out of product design/mechanical engineering, but I just haven't seen a good proof of concept yet. It always seems to be some variation of "Design a sphere. Oh wow, this is incredible!"