Doesn't the barrel remove this texture completely with the spin groves? Seems more of an issue than rough handeling.

Once the box magazine is loaded, it's not like the individual cartridges are being handled heavily.

Deer season involves a lot of loading and unloading a rifle (for those of use without removable magazines who are also bad at finding deer), and the bullets don't come out of it appreciably worse for wear. And that's for lead-free soft annealed copper ammunition. If you aren't being aggressively careless with your ammunition, it's not getting a lot of friction and scratches.