alt Hacker News"Fix this code" should ideally solve entire vulnerability classes, not just spot fix buffer overflows one by one. Thus it may be possible to design an LLM which can solve entire vulnerability classes and remain useful to users, but refuses to reason about specific buffer overflow vulnerabilities or specific race conditions, etc.
For example, "fix this code" on an ageing monolithic C codebase that accepts media files as input and outputs them visually to a display server could:
1. Recreate the software using a modular and loosely coupled architecture rather than monolithic and tightly coupled software architecture. For example, command line argument parser is a separate process, file format parser is a separate process and display server output is a separate process. If new features are added in the future (such as filters for manipulating output) then the architecture supports such additions with ease.
2. Use operating system sandboxing features to restrict what each modular component of the software architecture is permitted to do. Now that the parsers are separate processes, it's easy to pass an open file handle to the file format parser and only permit the process to read the file handle (not write to the file, not open any other file, not read the system clock, not open a new network socket, etc). The worst case impact of a parser bug is now significantly reduced.
3. Convert at least critical components to "safe" programming languages (Rust, Ada, SPARK, etc) which can be used to remove entire classes of bugs--read/write out of bounds, division by zero, numeric overflows, etc. For cryptography code--use a formal mathematical proof language. With a modular and loosely coupled architecture, different programming languages can be used depending on the use case--for example, assembly for video decoding where performance matters most and sandboxing can provide the security guarantee, Rust for implementing multi-threaded servers where race conditions must be avoided and Python for low-criticality user-adjustable code/plugins where ease of use and maintainability is most important.
4. Ensure software components are reproducible during their build.
5. ...etc
However, a prompt of "Are there any buffer overflow bugs in this codebase?" or "Fix the integer overflow vulnerability in add_numbers(x, y)" would be rejected. In the later case, telling the LLM to fix some specific bug in each of function1 through function9999 would force an LLM to reveal whether it thinks a bug exists or not. Responses of "Silly human, that bug doesn't exist in function596" or "Good find human, I've fixed that bug in function596 for you" allows a human to quickly narrow down where the LLM thinks a bug worthy of manual human detection can be found.
Replies
> "Fix the integer overflow vulnerability in add_numbers(x, y)" would be rejected.
This would make these tools completely useless. They aren't deterministic enough to give vague prompts like "fix this code" I'd prefer to be very explicit when using AI assistance to keep the scope in check for what I want the agent to touch.
It's MY agent, not someone else's. I don't want to auto rewrite in rust, refuse prompts against my own codebase (or someone else's, actually, if I'm working on open source), etc.
"Are there any buffer overflow bugs" is a perfectly valid prompt and in no way should ever be rejected by safeguards.
At that point, might as well just remove software development entirely as a use case and publicly state so "Due to safety concerns, agentic software development is no longer a valid use case" because other wise, what's the point if I can't be explicit in my prompts for both what I am looking for and what I want the LLM to do.
I'd be pretty pissed off if my LLM told me the only solution it'd be willing to implement to fix my code is to rewrite it in Rust. No way I'd pay for a model that refuses to fix bugs in the language given, especially because maybe I might not have the ability to convince other stakeholders to change it.