Wow, oddly similar thing happened to me. Anecdotally, I was working with a non-developer. She was building a reporting dashboard with Claude. I pointed out that the system she built was using the filesystem to store one line notes about the report, and I said "I would probably tell the agent to use a database for that, but it's not the end of the world if it works". And she replied "Claude said the developer's right, a database would be a better fit..."

So, a non-developer, who has no understanding of the underlying systems they are working with, fact-checked a developer with years of experience.

I'm not against fact checking, but it absolutely felt like a punch in the gut coming from a non-dev. There was no "trust" there. She couldn't rely on my expertise, she had to go back to her "source of truth" (Claude) and asked for a second opinion on something that she wouldn't be able to verify.

This is why collaboration exists. I trust my colleagues to give me insight+advice on things that I do not know. I don't ask AI to vet their decisions, because at some level there are judgement calls, and I WANT to trust them because it makes my life easier.

Treating AI as if the output is factual is just a misuse of the tools, I think. It's significantly more effective when given expert direction and the output can be verified by... an expert.

This resonates a lot with me. I'm the CTO of a software factory-ish company. I get daily emails from clients running our decisions through LLMs and asking for ridiculous stuff that would multiply the implementation cost for them, just because Claude said so. No, your 3 users app doesn't need to be SOC2 compliant. No, of course we don't have triple zone redundancy while you're paying a 50 USD AWS bill.

Most of the time, these are just concerns triggered by chatting by an over-zealous LLM; in the worst cases, they are demands.

I'm usually able to clear concerns and disarticulate the LLM by explaining to customers how much more expensive everything would be if we did things like that. But it feels so frustrating, it's like you have to prove yourself everytime and justify every decision. These interactions have made me question my future in the industry, if I'm willing to keep dealing with these situations. I'm trying to foster patience in my life to cope with this.

Yup, the arrogance when barging into unfamiliar domain that was previously reserved mainly to physics grads seems to have spread everywhere. I also had people opining on my expert area via their LLMs and the problem is they can't even ask the model the right question, let alone evaluate the nuance of their answer.

I also work at an intersection of tech, applied research and science. My experience has been different. It's been surprising how controlled my collaborators/coworkers have been with their reliance on AI.

Sure, they use it a lot, but despite the stereotypes against physicists in some of the replies to you, they have shown a good ability to catch themselves before leaning too hard on what an LLM tells them about things they are not experts in, and I haven't had any interactions where I felt like I was just arguing with a meat proxy.

Maybe it's because in the environment I'm in; it's relatively easy to just ask someone who is an expert in the topic for their advice.

My feeling is that this kind of overconfidence outside of one's domain is largely a thing for people with little "physical reality" experience. When all you deal with is the very flexible digital world, it becomes very easy to ignore how deep the knowledge and intuition goes in things that are directly constrained by reality. I consider myself to have been in this category too (CE background), though I have been making efforts to improve.

What do you suggest these people do instead? I’ve been frustrated by this recently: I pivoted to a new subfield and am working on stuff that I would love to dive deep into and really learn what is going on so I can speak intelligently about the tradeoffs etc. But that would take a long time, and I have tasks that I should get done. So I have found myself working with a pretty vague understanding that, when pressed by coworkers, quickly finds its limits. Then I go back and try to deepen my understanding enough to cover those limits. But because I’m not working with each detail of the problem, implementing line by line with time to think about what’s happening, there just isn’t time for me to learn this unfamiliar topic. But I would love to, and I would enjoy the work much more if I could. So what am I supposed to do?

Idk, seems like overconfidence outside your domain really got going in the '00s and is largely perpetrated by SWEs.

Something about making a lot of money and living in a world of abstractions really seems to fuel our sense of overconfidence in our abilities in other areas.

"I also hear a lot more questioning people in their domains and challenging opinions, then hearing what I can only imagine are fragmented pieces of conversations they had with an LLM thinking through some argument. Then there’s silence when you discuss shortcomings, then they come back later with their memorized fragments of what you said, combined with memorized fragments of the LLM response to the argument."

We hired this guy recently.

> It’s making me want to be a lot less collaborative with such individuals.

I have the same conclusion and am getting a growing list of mental blacklisted people who I just sort of ignore or greatly discount their efforts.

I suppose I always did this if people were time wasters but what used to be a tiny list of “idiots” is now getting much, much bigger. Hopefully it doesn’t grow to everyone I work with like an AI Nothing.

But isn't there value to refuting what Claude is saying if it deserves to be refuted?

The way I see it, if I was repeatedly getting sophisticated but subtly wrong arguments about my work that require me to understand why said argument is wrong, that's essentially drilling down to specifics of what precisely needs to be true to solve the problems I am aiming to solve. There is enormous value to this precision, isn't there?

I saw this happen a bit earlier this year, but most everyone I work with has learned that it was foolish. The feedback that they are doing something wrong needs to be explicit and strong. We're all going through a learning curve and establishing cultural norms is important at this time.

Isn't this basically the WebMD effect migrating to other domains?

feels like the same lament doctors have had for ages after anyone could google their symptoms then self-diagnose.

While AI is driving it the foundational cause feels like people having easy access to data/opinion that they trust but don't fully comprehend (or have bias towards). People do this in meat-space too, will confidently regurgitate garbage if they were told it from a person they trust as an expert.

Then there are the megacorporations of AI whose highly paid tech support folk happily copypasta your reported issues into Claude or Codex and paste its "solution" into an email as if you couldn't do that yourself and they don't even bother to check if the "solution" even works.

> It’s making me want to be a lot less collaborative with such individuals. I don’t want to sit around and refute Claude text outputs all day.

There are also the guys who are just blatantly meat proxies for Claude. You send them a message, and you get back a response that's all Claude (and disorganized and not really making sense to boot).

I've got one on an affiliated team and I basically don't want to collaborate with him at all anymore.

But you know, "AI is the future of work," and all that. Those guys get a pat on the head by higher-ups and probably think they're doing what they're supposed to.

I think there will be a rebalancing, and it might painful first for some. Confidence in LLM for highly complex tasks, very lots of context, most of all when this context isn’t in a single place or simply isn’t digitalised, will go down.

But right now, it’s still the shiny new toy

Yeah, I've given up on arguing with heavy LLM users. They're just a meat proxy, and I'm arguing with the LLM. No thanks.

We are going back to philosopher conversations. Just a few guys sitting on the stairs, with chalk, the street as their whiteboard, the jammer keeping the LLM-zombies away. After 2000 years, after the loudness makes right post-modern-pre-llm drivel of the frankfurter school- its a philosopher renaissance as resistance.

> I have people who have limited experience with software pushing out layers and layers of abstracted code that’s fairly sophisticated but often misguided in intent who will say what they’re doing is correct, with conviction.

Oh my god, physicist code is evolving!

I try to do the opposite. However, I do battle test ideas against LLMs a lot, as well as humans. The experts have very limited access and availability:

https://magarshak.com/blog/why-im-confident-in-my-views/

The LLMs are trained to often be more pessimistic off the bat than the humans. But you can wear them down with arguments and they change their mind.

Here is an example where I have pushed my ideas in areas where I am not an expert, and generated papers to submit to conferences in order to get them peer reviewed BY experts: https://magarshak.com/papers.html

This is exactly what you’re talking about, except done very carefully.

I think it is amazing and it will democratise information. A lot of times the concept ends up being simple with a lot of jargon exists as a gatekeeping method. Well of course the incumbents wouldn't like that rando from outside gets to understand and speak about things they worked on for many years. Sure he doesn't get it right 100% but he's in the correct direction.

Instant armchair pundit: just add tokens!

it's hard to gauge this take without specifics. My experience with LLMs tends to be towards the opposite concept; LLMs dissuading me of hunches and notions I have about things (where I have no particular expertise; societal-level things), saving me and others time and strife having an argument about something they were actually right about all along, as my internal doubts that I'm too embarrassed to bring up (because these things aren't my field) are confirmed as incorrect.

I'd be curious to know specific examples of LLM-generated advice that goes against the advice of experts and does not consider tradeoffs. I've not had this experience myself.

If I did have this experience, someone spouting off obvious LLM points that contradict my expert opinion on something, I'd be headed right over to gemini/claude/whatever to see where that's coming from. Not any differently than if someone cited a google result that contradicted my own experience.

[dead]

At some point, LLMs will be a far better source of truth, and then this distaste of people “outside their domains” is really just going to be a sort of snobbery from people who have had experience in a domain for a long time, (but they still have the same level of knowledge and insight as a person who just used an LLM to research).

These people are basically nascent “human supremacists”, who believe that only raw human insight and output has value, and is even superior, than equivalent output looked up and synthesized via LLM.