> people also expect those files to still be accessible i.e. if the agent invokes "make", which makes it impossible to solve perfectly

You could always use setuid to allow the agent to run designated commands whose operation depends on the files, without the agent itself being able to access the files.