At every UX job I've had we generally ran usability tests with 6 participants. This was before the online usability testing tools proliferated and we had a lab in our office (one way mirror with observation room and everything). Every UX researcher I worked with said based on the available research and personal experience it is as you said: more than 5-6 participants would hit diminishing returns pretty hard and wasn't really worth it.