> If we're disregarding shifts and adds as completing in negligible time

When considering multiplication algorithms the parameter N is the number of digits of the two numbers. In that model adds do not complete in negligible time, and instead take O(N) time.