The way caches are managed in CI is bonkers (at least in GitLab, but GitHub probably does the same). I guess it's built to conveniently work with the simplest of projects. I had to reimplement the mechanism by hand just to get basic things working, like not downloading the entire cache when only a few entries will be used for each build