I've seen behavior like that when the model wasn't being served with sufficiently sized context window