Colloquially, I do feel that "Wall Street" is strongly associated with the banking system that people touch regularly, and not boutique funds and offices that they've never heard of. Might be a jargon issue.