Correct, but it's been ages and the default actions/setup-go github action still uses go.sum instead. I see that someone already commented on the longstanding issue to reference this post, and that there is some hope that they'll update it!
The private repo mention in the comments there is kinda a good one, unfortunately. If someone runs a private gosum/goproxy (relatively common) and amends a tag (hopefully very uncommon but I have personally seen it happen at least three times) then the cache could be wrong because go.mod didn't have to change. Which is Bad™ but it depends on what edge cases they want to handle automatically, vs optimization for the majority case.
For well-behaving/stable/consistent setups I fully agree though, go.mod is both sufficient and better, and those other cases can probably just key off both instead. I think I've seen go.mod to change without go.sum changes (change an unused transitive dependency into a direct dependency), which can lead to your build needing something that wasn't cached because it was pruned in the previous version.
At this point I'm just assuming that all of the official actions under https://github.com/actions are unmaintained garbage until proven otherwise, given that high-impact issues like this can sit for years without as much as an initial triage: https://github.com/actions/delete-package-versions/issues/16...