One of the points of Unicode was to replace every other existing text encoding. This requires enough fidelity to be able to round-trip the text back and forth to them so that you can receive data in another text encoding, store it in Unicode, and then spit it back out in that same other text encoding without anything having changed.

The classic IBM PC text encoding ("codepage 437") already contains the card suits, gender symbols, and box drawing characters which are not text glyphs, so any "non-text symbols" battle was lost before it even started.