Well yes, in practice color often is chaotic. Nobody is color-coordinating the cars in the road, or the houses on a street, or the signs and advertisements and billboards. It's a free-for-all that turns into garish noise.

And more neutral environments with accent colors makes sense because the main accent is always people and their clothing. Your patterned red dress won't clash with a neutral background. It will likely clash with a patterned orange wall. A more neutral environment allowed for lots of colored accents to exist without competing or clashing with them.