Брюки цвета кофе is natural in Russian. Pretentious, but still natural.

It would make your Russian more posh, eccentric or sophisticated, depending on the context, but not necessary unnatural.

actually you will. "coffee color" it's distinct from brown. And then there is also "coffee with milk" color.

Won't be surprised if there is "pumpkin latte" color nowdays.