I remember having this issue ALL THE TIME with Gemini CLI but personally I haven't experienced that yet with agy.