alt Hacker NewsHow we measured AI writing across arXiv, and where the measurement breaks
Comments
I scored the full text of 12,750 arXiv papers from 2021 through 2026 to find out how many of these get flagged as machine written and how much it increased since the release of chatGPT. I purposely tuned the detector to avoid false positives. My detection rate pre chatGPT is around .4% for that reason.
The biggest results: in Jan of 2026 about 39% of papers got flagged as AI written. In computer science speicifcally the peak was at 65%. Mathematics barely moved away from 0.7%, though the proof heavy math texts might just not get picked up by the detector properly.
All this is a detector estimate of a statistical signal and not a proof any given author used AI. Machine written can also mean heavy AI-assisted editing.
There some real game theory mechanics at play when it comes to LLM usage in corporations. Devs are cranking out superficially superior code and documentation by just aiming the Claude Code fire hose at everything they can. Leadership encourages this because from what they can tell, there is no downside.
It's hard to say if this code is structurally better or worse than before, but it's certainly voluminous and as far as leadership could ever tell, with their flawed metrics, that is all that matters. It will be years before we figure out if this is a good idea and worth the cognitive atrophy.
Anyone not using LLMs all day is just not going to be as prolific. I can't imagine that the same factors aren't at play in the scientific research community where it's all about how much you can publish.
As with all text-only AI detection schemes, I am concerned about the accuracy of the detection. I'm skeptical of the methodology, specifically the final join of the three detector scores---how can you be sure that that final step does not introduce any biases? There's no source available, so it's difficult to tell exactly how this works or reproduce the research.
I've also uploaded text samples from my own (unreleased) research from pre-LLM era, and it's seemingly scoring pretty high on the LLM-detection scores. On other papers, nearly every sentence is highlighted as red "machine-leaning," but that does not impact the score? Additionally, there are dramatic differences between the scores for identical text with and without LaTeX formatting, despite the fact that it should not matter.
The takeaway from this should be "it is difficult to detect generated text and we should be careful about accepting results simply because they confirm a hypothesis."
--
Relatedly, the text above scores as highly machine-written, despite the fact that I just wrote it with my human hands, I promise :)
> If a tool marks 40% of new papers as machine-written but also marks 20% of papers written before ChatGPT existed, the real story is the 20% nobody mentioned.
When 65% of the papers you read have the characteristics of being AI written, whether or not you use AI to write, your writing will be influenced by the AI style. I imagine this must be particularly the case for newbie researchers who are still developing their writing style
I've looked at this same problem in an academic environment and have come to conclude that there is no way to reliably detect AI writing using only text. The reason for this is that no detector can take two identical inputs and classify one as synthetic (LLM) and one as organic (Human) and this situation can easily happen at the sentence or even paragraph level. Posed as a question, if a human writes a paragraph that happens to be exactly the same as a paragraph written by an LLM how do you classify that paragraph? There are a finite number of words and a finite way of combining them within an academic setting/field which means that we can't build a perfect classifier with just text. In practice, there are obvious tells and LLM-isms but these also change with time and each model has different tells so that even if a paper is full of LLM like writing we have no way to disambiguate between organic text that appears synthetic or synthetic text that appears organic.
A more interesting question, to me, is looking at a corpus of essays and analyzing how writing has changed with the introduction of LLMs. We can look at changes in vocabulary, linguistic features, style embeddings, regular embeddings, typos, errors, and references over time. When looked at in this way it is clear that academic writing has changed at the population level but what has led the change is harder to track down.
I am not a native English speaker. This is not surprising to me. I think most of the papers we write would be flagged by AI detectors.
It is not because we ask LLM to write us a paper about X. It is because we are bad at writing in a scientific style, and american editors expect us to do it. With LLMs, we can write in basic sentences and tell the LLM the idea and it converts that to nice writing.
If you write each paragraph and have an LLM make that paragraph more scientific, it is entirely your paper, but it is flagged as LLM generated. If you have an LLM write the paper but speak good enough English, you can make it look human even though it is not human.
Just to play devils advocate. These kind of papers are very verbose and boilerplate. I can imagine using AI to write 90% but then the actual novel content and explaining what’s important could be handwritten.
Perhaps that’s what’s happening.
> Here is the method, the results, and an honest account of the limitations.
This paper was written using AI, to be honest
One challenge with this approach is: could it be possible that the detector is simply learning to recognize words and jargon used more in the literature post 2022 as 'AI'?
For example, LLMs love to talk about LLMs (and the people who write with LLMs love to write about LLMs). Could "large language model" itself therefore be flagged as an AI-like phrase by this approach? It didn't exist much in the literature before 2022, does now, and certainly does more in AI-generated text: but, it is not actually a great way to distinguish modern AI generated text from human written text.
A helpful control would be to show that on some cohort of papers that can be declared reasonably clean of LLM generated text post 2023 there are very low rates compared to the arxiv.
For example, while papers in the journals Nature and Science are unlikely to be entirely LLM free at this point, if those were tested through 2026, we should see a line significantly lower than the arxiv's growth.
I tried the detector on several of my own Stack Overflow answers, and the fourth or fifth one I tried was flagged as "85 % machine / likely machine-written" with the default settings, well above the default sensitivity threshold of 42%. When I turn off the option to strip LaTeX commands (the source is Markdown), that jumps to 96% machine.
Admittedly, a Stack Overflow answer is somewhat outside the realm of scientific writing, so it's still possible that the detector may be accurate within that domain. That said, it's a cautionary tale on the hazards of applying classifiers like this outside of the domain that they were trained on.
I work with several researchers (primarily working on consensus protocols etc).
We were discussing research in general and I asked them: "Do you prefer the writing of the papers or the research?"
They, almost unanimously, agreed that they preferred the research. This makes sense as if they preferred writing they probably would have chosen another profession.
I say this b/c having LLMs available to turn research diagrams, code etc into a paper (or at least the starting point of a paper) will probably lead to MORE quality research papers. This is b/c I'm sure there was some friction in a researcher's mind of "I would love to do the research on this but don't want the trouble of writing the paper".
Put another way: on a 2D plot with one axis being the skills as a researcher and the other being hatred of writing, LLMs may "unlock" the people high on both axes to get more papers out.
Post Script: I agree that this could also lead to more BAD papers but the net may turn out to be positive in the long run.
Something I've thought about a lot is that there is having someone with some domain knowledge or reason to care a lot about a particular issue spend a bunch of tokens and cycles on it until something useful comes out the other end. The most obvious ones are the math problems that have been coming out and help push the frontier of various areas of math. Another example is taking all of the public NYC open data ecosystem and crunching it to get some value which I have spent a lot of time and tokens on but not found a great medium to share.
The question is just how to organize these outputs and conclusions in a way that is consistently reproducible and also how to correct errors or remove LLM nonsense where it refuses to take a position on something.
Before it made sense to do this in papers but it feels like we need something like a paper format.. that is fully reproducible ideally and optimized for aggregating knowledge in a better way. I.e. before a person spent months on one of these and there was just more filtering, and the output itself was a clear signal of time spent and effort that no longer exists.
I get 0% (accurately) on my latest paper. Not super surprised, as I intentionally avoid some LLM-isms that I used to use because I don't want reviewers to have even the slightest indication that text is LLM-generated (even if in principle I'm not opposed to polishing or even wholesale generating academic text if it can convey the original research well, especially for non-native speakers).
I don't think the problem is as bad as a naive reading of this article suggests. I'm highly skeptical that anywhere near 65% of recent CS papers that I've read (mostly systems papers) are substantially AI-written. I threw some recent papers I've read into the system and they come back as 0-7%.
To corroborate this - I found pretty similar rates of AI-flagged papers over time from running pangram on ArXiv (abstracts) in a physics subfield, which is currently at about 25% as of April (write up here: https://peterse.github.io/2026/06/15/The-rising-tide-pt1.htm...).
Its not clear from your writeup what threshold needs to be reached to be classified as "machine written". A preprint where half the text is human and half is 100% AI should be a different category than a preprint where 100% of the text is AI-assisted.
Also its cool that you're making the detector available. When you say "cheap to run", do you know how this compares to pricing for a commercial detector pangram or GPTZero?
Is it just me, or does this article itself read as AI generated?.. Ironically, I tried running the text of this article through their own classifier, and pretty much all of it was highlighted in red. I'm not sure what to make of this. Is this some kind of intentional irony?
I run most professional statements and articles through LLMs before submission. This helps correct grammar and improves accessibility through better syntax (because I write exactly how I think).
The article doesn't seem to mention consideration of AI for polishing human work.
I just generated some docs for a lib I'm writing and it says: > 9 % machine likely human-written
I’m very skeptical of these results. I run a pretty large group paper website on top of arXiv and we run pangram on papers. The numbers are not nearly this high.
One thing I see a lot is papers flagged as AI because they include llm rollouts in the paper as examples.
I ran two papers and three blog posts of mine through here, and all but one (correctly) flagged as 0-1% machine written. The other (human written) blog post was 20%. Pretty good afaict!
On my knees begging that the developers of AI detectors use them on their own writing and that at least pretended to mask the obvious LLM cliches in their blog posts.
Not convinced that this slop measurement is useful.
I asked Codex to generate an article with a high score and then asked Codex to (reverse?) hill climb that score. The original generation scored 97% and then the optimized one scored 1%. Both are pretty bad and read like slop.
https://gist.github.com/wbew/8a2bd6686bf875210f2244ac8ea65bf...
Awesome work, the detector seems accurate on a bunch of texts I tried, surprisingly even human/ai hybrid texts
the funniest part of these AI detectors is that if I were to upload any of einstin's paper's they will all be flagged as AI-written. it makes sense because it's part of their training data.
but this post makes me wonder, if more papers' are written with AI, or the shape of knowledge of converging?
the measurement breaks because the detector is probably also AI, and at that point you are just watching two language models argue about who wrote what
Good contents should always thrive, with or without AI
> and an honest account of the limitations.
Gotta be trolling :-D
> Here is the method, the results, and an honest account of the limitations.
Pot meet kettle?
People are saying this is a bad thing but is it really a problem? The compelling aspect of research is the data and/or description of work, not the writing. Papers probably should be written by AI so that they're clear and well presented, while the researchers should focus on generating good data. If there is no data or work behind the paper, we should question whether the research group needs funding.
No wonder reading all these paper is tiring these days.
I don't want to read slop generated by AI. AI written articles are generally low effort.
What's the point of looking when no one even cares?
consider the fraudster that went around suing people on the basis of his absurd claims of being bitcoin's creator. He's now transitioned to using AI to gather graduate degrees and is obtaining masters and doctoral degrees at a regular place and writing multiple 'papers' per day that are all quite obviously AI slop.
People report his cheating and publications and simply no one cares... and this is someone court adjudicated to have fabricated evidence in court on a massive scale, including through the use of AI.
But when it comes to the degrees and publication everyone involved that wanted paid got paid, and apparently that's all that matters.
Honestly academic writing is the only place where I think AI slop might be an improvement over the status quo writing style.
The important question is:
So what?
Genuinely. I get that there may be some visceral reaction against this, but when I break it down, I mostly fail to see the problem. Seems like what is actually important is:
Compared to before, when a human reads it, do they -- or society -- get something good out of it? Is it worth it to add this to the "pantheon?"
If that's not what's happening enough, and if this doesn't describe the process -- then the problem lies elsewhere, no?
[dead]
[dead]
[flagged]
[flagged]
Good contents will always thrive, with or without AI
I'm not sure what to think... I uploaded a PyHPC workshop paper I wrote in 2011 and it said 27% machine.
I also uploaded my PhD dissertation from 2012 and got back 40% machine, which is just barely below the 42% threshold.
I don't publish anymore... but does this mean I wrote like a LLM or did LLMs learn from me? :p
Update: I also uploaded a IEEE CLUSTERS paper I wrote in 2015 and it came back 74% machine written :|