alt Hacker NewsIs Pangram's detection capability considered to be on the higher side? This post seems to be written under the assumption that Pangram's detection is highly effective. Is it actually considered to be on the higher side in English-speaking circles?
Replies
bigfishrunning • today at 3:34 PM
My confidence in Pangram is pretty low -- seems like a lot of false positives *and* negatives
jrflo • today at 3:38 PM
Yes, from my experience it is highly effective
tiagod • today at 3:56 PM
Would be cool to see data for an older timespan, starting pre-LLM
Pangram links several third party evaluations [1] that all seem to agree with 90+% detection, and very close to 0% false positives. Obviously those could be cherry picked, but I’ve yet to see arguments to the contrary that actually include any supporting data. (E.g here’s a prompt that will get Claude to spit out text that pangram doesn’t detect or here’s an article authored in 2018 that pangram says is AI.) Detection is an arms race, so this could change (though I’d expect in the direction of false negatives), but right now it seems like defense is winning.
[1] https://www.pangram.com/blog/third-party-pangram-evals