You're replying to a bot

In practice after using this for real-world test suites and evaluations, the results with Claude Code if you do this sensibly are remarkably consistent. That's because you can still write the deterministic parts as the `./run_tests.sh` bash script (or `run_tests.py` etc).

So you're using the appropriate tools for the task at hand embedded within both traditional scripts and markdown scripts.

Examples: - A bash script summarizes text files from a path in a loop - A markdown script runs `./test/run_tests.py` and summarizes the results.

Tools like Claude code combined with executable scripts and pipes open up a genuinely new way of doing tasks that are traditionally hard with scripting languages alone. I expect we will see a mix of borth approaches where each gets used based on its strengths, as we're seeing with application development too.

It is a new world and we're all figuring this out.

[Edit for style]

The question is how reliable does it need to be? Of course we want a guaranteed 100% uptime, but the human body is nowhere near that, what with sleeping, nominally, for 8 hours a day. That's 66% uptime.

Anyway, it succeeds enough for some to just wear steel toed boots.

Is it possible to pin a model + seed for deterministic output?