Pollen can be broadly separated into airborne and not-airborne.

Ragweed pollen is light enough to be borne miles by wind. Goldenrod pollen is too heavy for that, but is sticky, and is carried on fur and feathers. Ironically, people blame the showy goldenrod blooms for allergies, although they likely have never had goldenrod pollen in their breathing passages - while lowly, hiding ragweed unleashes millions of barbed pollens spores upon their breathing passages. (Ragweed flowers are small and green - you can stare at a plant and not realize it's blooming!)

Likewise: the sap of poison ivy is strongly allergenic; the sap of maple trees almost never, due to reactivity with immune systems. Americans are likely to be exposed to both.

So, in short: there are plants that are potential allergy-sources, and others that are not.