I think comparing assembly with arithmetic is dead wrong. Arithmetic is something that you use constantly in virtually any mathematical activity you will ever do, at least at the under-graduate level. There is literally 0 calculus, statistics, or algebra you could understand if you didn't know arithmetic.

In contrast, you can have a very successful, very advanced career in computer science or in programming without once in your life touching a line of assembler code. It's not very likely, and you'll be all the poorer for it, but it's certainly possible.

Assembly language is much more like learning the foundations of mathematics, like Hilbert's program (except, of course, historically that came a few millenia after).