Interacting with anyone is a risk. They might take any form of attention to a level that forces you to set a boundary, or uncomfortably accepting the boundary already being crossed, turning a positive intent into a a net-negative interaction. If you're attracted to someone, there are a lot less boundaries. If you find someone unattractive, a lot of other people probably do to, increasing the risk they're attention starved, and more likely to make it awkward. Most people feel that risk, regardless of their own attractiveness. "Confidence" can help signal they're less likely to make it weird, but it's certainly no guarantee.