You need to know the law. That is being careful, but it doesn't mean it's always illegal to photograph children in public.

At least on the street and in sports my experience is people using purpose-built cameras get harassed.

People using cell phones as cameras get a pass (at least in sports).