logoalt Hacker News

Youdentoday at 5:34 PM0 repliesview on HN

I think that's reasonable. The goal of the benchmark is to determine how the model performs on real-world tasks. If real-world tasks trigger Anthropic's classifiers, that's a failure on Anthropic's part.

I feel that choosing tasks that don't trip the classifier would also be a form of bias towards Anthropic.

In case it's not clear, the coding tasks are really benign things; there's nothing security-, health- or biology- related in there. The classifier being tripped is definitely unreasonable.