Disfluencies are not necessarily "filler". They can convey mood or hesitation. Cutting them can change the meaning.

A trivial example is "umm... well... (sigh) okay" versus just "okay". Not okay!