"Брюки цвета кофе" ("pants of coffee color") is natural, "коричневые брюки" ("brown pants") is natural, but "кофейные брюки" is not. In fact the latter would likely be interpreted as "coffee pants" or "pants made out of coffee."
"кофейные брюки" is totally ok. everybody will understand it.
it's just the way the russian language is. you can abuse it, you can come up with words that do not really exist in language and make no sense, yet, everybody will understand what you meant to say
"кофейного цвета брюки" is acceptable too.