RLHF is an increasingly small part of training though? From what I understand most of the capability gain is in RLVR

Nah, the last few generations have more RLVR in the data mix. Which is more CPU intensive and very much amenable to the bitter lesson as you can reduce the loss by doing more rollouts in your tool environment.

rlhf = reinforcement learning from human feedback

(had to look it up)

More RLHF is in fact scaling.

The scaling with reasoning models is more and more with things like verifiable rewards (coding and math), in line with bitter lesson and also Sutton invented lots of modern RL.