That's a pretty poor straw man - the issue is the amount of harm caused, not that there is a potential for some minuscule amount.

Also we need detractors because if we race into any technological advance too quickly we may cause unnecessary harm. Not all progress is without harms, and we need to be responsible about implementing it as risk-free as possible.