CC-Canary: Detect early signs of regressions in Claude Code

A useful(ish) trick I've found is adding a persona block to my CLAUDE.md. When it stops addressing me as 'meatbag' I know the HK-47 persona instructions are not being followed, which means other instructions are not being followed. Dumb trick? Yup. Does it work? Kinda? Does it make programming a lot more fun and funny? Heck yes.

Don't lecture me on basins of attraction--we all know HK is a great programmer.

My attitude towards this is growing similar to my attitude towards Windows. If I have to fight against my tools and they are actively working against me, I'd rather save the sanity and time and just find a new tool.

I feel like asking the thing that you are measuring, and don’t trust, to measure itself might not produce the best measurements.

What is "drift"? It seems to be one of those words that LLMs love to say but it doesn't really mean anything ("gap" is another one).

Interesting approach, I've been particularly interested in tracking and being able to understand if adding skills or tweaking prompts is making things better or worse.

Anyone know of any other similar tools that allow you to track across harnesses, while coding?

Running evals as a solo dev is too cost restrictive I think.

the actual canary is the need for the canary itself

See also https://marginlab.ai/trackers/claude-code-historical-perform... for a more conventional approach to track regressions

This project is somewhat unconventional in its approach, but that might reveal issues that are masked in typical benchmark datasets

thanks

[dead]