What's the largest software project AI can complete on its own?

I have been using Claude to build a clone of Bash in Rust (https://github.com/jdstanhope/huck). It has been goin going on now for 81 days, 2600 commits and I have covered a good chunk of the features such that I can source by .bashrc with a number of extensions. It has been doing pretty good but it will sometimes spend hours to do few line fix and other times pushback on making big changes despite have over 3000 tests.

That being said, I am still guiding it on a regular basis and only recently let it run open ended trying fixing small to medium size bugs. The next stage will be try let it run in a loop trying to fix the 100 issues it has identified as well as getting most of the bash test suite to pass.

"Reproduce an existing piece of software" doesn't seem like a problem that generalizes well to "develop a new piece of software," for a fair number of reasons.

I'm experimenting with large-ish software projects using Claude Fable. The results are surprising to me.

- Project portfolio management software. Good results so far. This is unguided experimentation: Fable researches feature ideas and plans them, then Opus 5 implements them, without me in the loop. The features include things such as an auth server, full-text search, audit logging, resource leveling, HA/DR, IaC, and more.

- FHIR & SNOMED medical software experimentation in demo-only environments. This has mixed results so far. This is guided experimentation: the FHIR specifications and SNOMED specifications are thorough and I'm in the loop. Fable plans OK, but Opus 5 is thrashing a lot, going down many rabbit holes.

I think the best part about this benchmark is that they finally stopped pretending you can evaluate a complex engineering task with $ 5 worth of inference. If a task takes a human weeks, you need to give the agent a comparable search space to find solutions

We (Team of 3) built a retrieval engine from scratch in .NET for investigative search, with AI doing most of the backend implementation while humans handled the design and architecture and front end. It was developed using a task based, incremental approach. Includes more than 3,000 tests and extensive documentation, and is optimized for asynchronous ingestion using NATS and a disk backed WAL.

It includes a custom WAL, structure aware semantic chunking, ingestion and indexing pipelines, hybrid (rrf), text and vector retrieval, ACL aware filtering at query level, query classifier, citation tracking, and a CrateDB backend that performs fused retrieval over a single table. total LOC: 180 K. Status: evaluation by State agencies for investigative search on legacy documents. tool used - Codex, Luna, and earlier codex 5.3, and gpt 4 mini.

I'm looking at these reports and then at my sessions with Claude. In latest one, it got stuck in a loop fixing one thing and breaking another for 10 iterations, until I've noticed and gave it the insight. Or where it started continuously writing "echo", I don't know for what purpose. And I ask how, how can you not be the driver; how can you let these agents roam free and actually accomplish things.

I believe they can make ANY software project the largest it can be!

This solution should look for more problems.

So I can not code. I just tell an AI the concept of what I want my software to do and execute in blocks. Most operations are well defined and depend on a central mechanism and that is where everything starts. The rest is like an octopus : the body defines the concept, the arms the consequences. This way there is no need for long code since every segment has a clear beginning and end. It is also quite efficient overall. My point is that long code can be circumvented by clever architecture.

This actually proves the hardest part of software engineering is defining requirements.

- I ll consider any of these models as god tier the day one of these 2 happen

- They build a browser engine from scratch that implements every whatwg spec for the last 15 yrs, fixes every bug chromium has and runs 10x faster than chromium

- they build a super highly optimized version of windows that runs much much faster than windows 11

It's smaller than an actualization of the Memex, that's for sure. It's been 80 years since it was seen as a good solution to the greatest problem facing science at the time.

80 years, yet we still can't mark up hypertext without copying and then modifications to the source.

This one is also very interesting: https://programbench.com/

i've been working on several rather large projects these past few months, and i'm trying to write as little code as possible.

I don't think i wrote more than 10 lines of code in the largest project i'm working on. Lines of code: Java: 900_635, typescript: 725_418, C++: 180_445, Dart: 96_181.

It's been obvious from the start that no model, as good as it is, can do large(-ish) amounts of work by its own without supervision, control, criticism, etc ... If left unsupervised, models usually do half the work, leaving stubs and todos everywhere.

Quality comes from applying software engineering principles as much as possible, just like you would do with teams of junior devs: planning sessions and implementation sessions with adversarial critiques, specifying as much as possible upfront, planning unit/smoke/integration tests, etc ...

Many systems rely on swarm of agents to build software but i've found it very difficult to get good results without lots of overhead/token waste because of inter agent communications mostly.

So instead i built what is mostly a workflow engine to structure / organize processes into workflows with different agents assigned different roles. I've setup a basic landing page here https://kodfactory.com if anyone wants to follow along.

> without access to the original source code

All models in the leaderboard probably have had access to the original source code in their training data.

my biggest so far was https://apps.apple.com/at/app/aifcc-ai-first-computer/id6782... a whole linux distro in a mac app. claude worked for about 2 weeks.

Seems like most of developers and humans vibecoding new projects with LLMs. But I can see they have problems in the codebase with AI generated code, or security breachs etc. Whats the limit of it? In other words, I'm curious about what the future holds.

> MirrorCode only covers programs with plain-text output. MirrorCode does not test AI abilities in software that is inherently visual

It’s an interesting eval but this is a distinct limitation — most applications that consumers use directly are not purely text-based.

whatever it is, it will be heavily skewed in favor of whatever there is a lot of examples of. AI could bolt together many cookie-cutter pieces into a very large YOLO system.

The trouble comes when you now want that system to do something unique for your business, and those ripple effects through the entire system you didn’t design become very painful.

I am curious how Claude Opus 5 fares - similar, better, or (my guess) worse than Fable 5.

Running a Loop on the well defined goal seems promising: https://www.youtube.com/shorts/EtxdB7gpFqY

Would be nice if they also tested other models instead of several old versions of GPT.

Couldn't find what exact tests they are running. The GitHub repo is very obscure to be read by my human brain.

It really depends on how you define "complete". It's possible the answer in "not large at all" if we're looking for high-quality, reliable, user-friendly software that works near perfectly after being written, tested, deployed completely by AI with no human intervention.

People who try to get AI to do a lot of un-manned work seem to have a bit of a bias toward viewing LLM work as "good" or "good enough" in a way that they wouldn't have accepted in the pre-AI era.

What size project an AI can complete on its own is IMHO not an interesting question. The AIs and harnesses are getting good enough that quite large things can be built, primarily as a function of cost.

The interesting question in my view is what is the largest system an AI can maintain over a long time horizon -- while keeping a coherent architecture that adapts and responds well to change. All the while remaining performant, scalable, and understandable to humans. I am unaware of what is the currently longest-lived vibe coded project but it cannot be more than a few years old.

So to my mind, we don't have very much real data yet.

I'm trying to build a competitor to Parasolid and... it's a struggle getting towards the end of the first 90% of the work. It's getting back to the point where I'm buying textbooks and reading papers because moderate domain knowledge is not enough to drive LLMs to completion here.

The space is too big and the code too extensive for a 1M context window and the capabilities of the current SOTA models.

Why just the commercial ones !??

I'm curious about that too.

I'm building a language with AI, and when I leave the AI running on its own, it keeps generating code in strange directions. I wonder what kind of structure AI-only code would have.