Angular momentum has energy x time units.

These unit equivalences have to be carefully interpreted. Like when things are multipled, are they in the same direction? Torque has the same units as work: force x distance, i.e. energy. But the force is perpendicular to the distance; it's completely different, and not a simple scalar value: torque is a vector with an orientation in space. Moving something against friction over 10cm, and using a 10cm bar to apply leverage, are entirely different.