I think knowing what you want (and being able to articulate it with adequate specificity, and detecting when processes drift away from it) is a real skill/field of expertise. I just don't think it's specific to interfacing with "AI."

> Been skeptical for quite some time that "using AI" is a real skill/genuine field of expertise.

I feel using AI (effectively) is not too far from the skillset of programming. It is still a machine following instructions (just, maddenigly non-deterministic, but still close enough), so the same insticts of breaking down work into clearly defined sequences that make a good programmer also make a good AI jockey.

In this case they didn’t know what they were looking for. They just wanted some new, non-trivial exploit.

For the vast majority of corporate usage of AI for SWE, you have a much better idea of what you want, or what the problem is, etc etc. And communicating that to the model effectively is absolutely a skill. I see colleagues every day who very much do not have that skill.

It can be a skill, but a very soft skill like keeping control of and educating 30 kindergarten students. It’s a skill, hard earned and nebulous. you can recognize it clearly when you see it, but it’s very hard to define/measure numerically and will never achieve anything close to “perfect” results.

Until these models become many factors more deterministic, at least. That’s sort of the hard barrier here, and given the underlying tech it’s a really tough one to overcome

It's still a skill for now. Prompt engineering specifically is perhaps not as much of a skill anymore, but I think there's a level of skill involved in understanding and/or anticipating when a model is going off the rails.

Some people struggle to effectively use AI because they either have to spend a lot of time reading and thinking about the response or they have a hard time noticing when the model is subtly going off the rails. Others use it to good effect because they can anticipate which tasks would be better handled manually, or are good at catching that the way the model is describing something subtly indicates a misunderstanding.

I think we still have a solid control lever on the quality of code we can get AI to generate. Using things like linters and code style checkers, as well as setting up the markdown documentation to guide the agent to generating consistent code will certainly generate different code than just prompting.

Similarly there was an example of edit: Terence (not Eric) Tao chatting with an agent attempting to solve a math problem. "Using AI" means applying your expertise to interact with it as you would a high level colleague. 2 experts in a field don't need to have perfect english and a bloated prompt, they have a massive education/experience common background to fall back on.

It does appear that anthropic in particular is attempting to create a more common experience across expertise levels, but in the current landscape an expert and a novice are unlikely to get the same results. But that does seem to be the goal...

Plus if or when they do work noticeably better, you'll be able to just ask them to improve your prompts. Because those prompts are discussed heavily online, and are in the training data.

You can do this now. It works alright sometimes. Other times you're reminded that this is largely just reading tea leaves, and you're trying very hard to separate anecdotes from data and not anthropomorphize it.

Literacy is the actual skill on display here.

You can make a model/agent as powerful as you want and it still won't be able to recover the author's intent if it wasn't even implied. Information theory still applies. No amount of parameters will change this.

Many of the AI development meetings my clients have sound suspiciously like writing or English classes. If a massive AI bubble is what it takes to get my team to communicate effectively, I'm all for it.

LLMs and their coding harnesses like CC, Codex, or such are a tool like git or bash or your favorite programming language. And like any tool, they have gotchas, weird concepts, and so on.

Indeed it's nothing hard to learn but there is a learning curve. E.g. knowing which model has which capabilities, and figuring out how to best manage context, permissions, worktrees, etc. There isn't one "right" way to use it but there are more efficient ways and less efficient ways.

It definitely was a real skill; there was a time when you could get impressive results out of frontier LLMs if you knew the most advanced tricks for prompting them, but not if you just told them "hey solve this problem for me".

However, it's not a sustainable skill, because the labs care a lot about making "hey solve this problem for me" work well, and so put out models that are better and better at working with unsophisticated prompts over time.

There's an excellent piece about this, but it's unfortunately paywalled: https://www.theargumentmag.com/p/can-you-tinker-your-way-out...