No, office hours shouldn't be removed. Perhaps professors should just not reward people who come to office hours beyond the extra instruction that is given. Eg no special knowledge communicated solely in office hours("this question is going to be on the test next week"), and no special treatment ("this student got the wrong answer but I know from office hours they're trying really hard so I'll give them some extra points")