logoalt Hacker News

strongpigeontoday at 3:23 PM0 repliesview on HN

You don’t normally compress the system prompts, though I guess maybe it treats its own summary with more authority. This article [0] talks about the problem very well.

Though I feel it’s most likely because models tend to degrade on large context (which can be seen experimentally). My guess is that they aren’t RLed on large context as much, but that’s just a guess.

[0]: https://openai.com/index/instruction-hierarchy-challenge/