logoalt Hacker News

popkkksidhd • today at 2:59 PM • 2 replies • view on HN

Even with developing, I’ve noticed understanding “beneath” where you’re working greatly accelerates LLM assisted and non LLM assisted work. I think that’s the true way to get good at using LLMs - any other “prompt engineering” kinda stuff is snake oil.

And yeah, I’m not sure how that goes away either - life is full of tradeoffs, how do you RL things with no objective truth?


Replies

FeteCommuniste • today at 3:49 PM

The paradox of LLM usage: the best way to use an LLM well in a domain is to know the domain, but if you spend all your time using LLMs, your domain knowledge may atrophy.

➕ show 1 reply
deltarholamda • today at 3:24 PM

Understanding "beneath" is a good idea. I used the free version of Grok to generate some Ansible scripts. I don't use Ansible often enough to be really adept, and I appreciated some help.

It was pretty good, understood what I wanted to do, and broke it up in playbooks/roles/etc. quite well. But then it did things like continuously overwrite the LetsEncrypt cert for each domain with the next one in a loop. It understood "this needs to be a loop" and how to optimize that loop, but not that it was going to overwrite each domain's key with the next.

I'm a pretty good sysadmin, so I recognized this and could take what it did and fix it, but it required a deeper understanding of the underside of nginx/LetsEncrypt and the filesystem to even see what it was doing.

Of course, once you run the playbook, test a domain and see its SSL cert is all jacked up you'll go looking for WTF happened, but just blindly taking the output as given meant it was horribly broken.

It may be that Ansible, by its nature, is a bit sui generis because it touches so many unconnected domains, and therefore is harder than outputting Python. I don't know. But one needs more than a surface level understanding of those domains if you want to catch the LLM being stupid.