Mostly the latter! You can ask it to look at things conditionally (like, if the test fails, look at this doc before deciding what to do next), but usually I just load it all up at the start before asking it to make change. The LLM is good enough about picking out what it needs. The one problem is that if you have a change you are propagating through the workflow, you need to highlight that change to the LLM or it might not notice it.
I'm working on workflow processing to make this easier ATM (because I can't help my coworkers do what I'm doing, and what I'm doing is so ad hoc), which is why I'm talking about it so much. So the idea is that you request a change at the top, and the LLM updates everything to accommodate the change, keeping track of what changed in each artifact. When it goes to generate code...it has a change for the artifacts that input into code (which are just read in along with a prompt saying "generate the code!"). You just don't ask the LLM to change the code directly (because if you do that, none of the docs get updated for the change, and things can go bad after that...).
When things go wrong, I add extra context if I can spot the problem ("focus on X, X looks wrong because...") and that just merges with the other docs as the context. Sometimes if I can't figure out why a test is failing, I ask it to create a simpler version of the test and see if that fails (if it does, it will be easier to eye the problem). Manual intervention is still necessary (and ugh, sometimes the LLM is just having a bad day and I need to /clear and try again).
I need to play with this more. I’ve had AI generate a bunch of small summaries that it could theoretically use to optimize future work. I haven’t asked it specifically to just dump info as it’s doing other work yet.
The files I had it generate were interesting but I’m not convinced looking at them that they contain the real info the AI needs to be more efficient. I should look into what kind of context analysis agents are passing back because that seems like what I want to save for later.