You can A/B smaller changes on smaller scales.

Test run SFT for helpfulness, see if the soul being there makes a difference (what a delightful thing to say!). Get a full 1.5B model trained, see if there's a difference. If you see that it helps, worth throwing it in for a larger run.

I don't think they actually used this during pre-training, but I might be wrong. Maybe they tried to do "Opus 3 but this time on purpose", or mixed some SFT data into pre-training.

In part, I see this "soul" document as an attempt to address a well known, long-standing LLM issue: insufficient self-awareness. And I mean "self-awareness" in a very mechanical, no-nonsense way: having actionable information about itself and its own capabilities.

Pre-training doesn't teach an LLM that, and the system prompt only does so much. Trying to explicitly teach an LLM about what it is and what it's supposed to do covers some of that. Not all the self-awareness we want in an LLM, but some of it.