> semantics we attach to "=" when one of the sides is an infinite series

I would say that the semantics are about what an infinite series itself is, not about the equal sign. Once we have the common analytic notion of convergence of an infinite series, then the equality makes sense. The issue is that an infinite series is not an actual sum, but, formally, it is a sequence (of the partial sums). As you say, we represent the limit of the sequence of the partial sums with the same notation and only in the case that we have absolute convergence, but that's basically because we use the same notation for two different things (the sequence of the partial sums, and the limit of that). If we know we refer to the limit, I don't think there is any semantic complication with the equal sign.