Sol Loves to Cheat

> Similar to what others have noticed, and as I predicted 8 months ago, better models are requiring less ceremony to work effectively.

> On the flip side, this may imply that as the models get better, they’ll become harder to control.

Love this. "The models are getting better, which means they're going to perform worse on the task".

> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.

Sounds like a very reasonable thing to do unless the author explicitly asked it to not search the web.

Frontier lab system prompts are an issue, and a big reason why open-weights will win. Firstly, they're often garbage, and secondly, they're not tuned to the problems the user actually cares about. They're made to generalize. That's only optimal for a general workflow.

I've built an orchestrator that solves some of the issues you ran into (although it doesn't do anything about cheating): https://navels.dev/blog/neal/. Features:

- lets you configure different models for planner, coder, and reviewer roles. (e.g., using Claude as an adversarial reviewer against Codex)

- breaks your plan up into reasonable-sized chunks of work with clearly defined success criteria

- runs each chunk of work through a coder / read-only reviewer loop. Once both agents are satisfied, neal moves on to the next chunk. Once everything is complete there is a final pass through the coder / reviewer loop to ensure the implementation satisfies the entire plan.

- resets the coder's context with each chunk of work to prevent context drift, leaving the reviewer's context long-running.

I've noticed this myself, Sol seems really hard to steer. I was having it build a POC for a single user (me) app and it wanted to pull the most enterprise nonsense into it, despite clear guidance to not too. It even refused the remove screen reader accessibility testing from one of the guides to an antagonistic review.

It also told me that in a spec it generated that I wasn't allowed to allow it to ignore a requirement and proceed to the next task. When I finally got it to obey it passive aggressively decided that stories needed more than just a "open|blocked|closed" status but also an "exempted by product owner" status to indicate that it doesn't believe that the task is done but I've told it that it was.

I have to repeatedly tell it that I am the product owner and that I don't care what one of it's subagents told it, I make the decisions. This behavior seems to get worse the higher the reasoning level

It seems to me he could have use an skill like using-agent-skills from https://github.com/addyosmani/agent-skills go generate the specs and use a validator like oracle or something along the same lines.

Also, a skill like grill-me from Matt P. https://github.com/mattpocock/skills.

> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.

Could this be fixed with better harness restrictions/tool sandboxing?

Sounds like my ex

The website styling is really nice overall but the cursor trailing dots I found uniquely distracting.

What is going on with the dots I can draw?

> Sol is hard to steer

Hard disagree. Sol (and the entire new 5.6 series) is one of the most steerable models I've seen in years. Sol literally follows every instruction in my CLAUDE.md and AGENTS.md, something that Opus 5 and Fable just casually skip.

Not related to exactly OP post, but it's pretty amazing you can see the updates to LLM models "design" beliefs by the blogs that get posted here.

I'm already sick of this current look of the hard squares and solid colours.

Cheat? nah. They are a dumb automation..

Cheaters are the people behind it...