alt Hacker NewsThat’s a really nice pipeline. The “save to file between steps” pattern seems to appear very naturally once agents start doing multi-stage work.
One thing I’ve noticed when experimenting with similar workflows is that once artifacts start accumulating (drafts, logs, intermediate reports, etc.), you start running into small infrastructure questions pretty quickly:
– where intermediate artifacts live – how later agents reference them – how long they should persist – whether they’re part of the workflow state or just temporary outputs
For small pipelines the filesystem works great, but as the number of steps grows it starts to look more like a little dataflow system than just a sequence of prompts.
Do you usually just keep everything as local files, or have you experimented with something like object storage or a shared artifact layer between agents?
In my prompting framework I have a workflow that the agent would scan all the artifacts in my closed/ folder and create a yyyymmdd-archive artifact which records all artifact name and their summaries, then just delete them. Since the framework is deeply integrated with git, the artifact can be digged up from git history via the recorded names.