Market harm is not required for something to count as infringement, but it matters for certain defenses and damages.

Simply writing new adventures for existing copyrighted characters is usually treated as creating an unauthorized derivative work. Writing Harry Potter from the perspective of the Weasley twins, for example, is not fair use.

Distribution is one part of fair use but it isn't the focus of it - fair use is a defense against infringement, but it's still infringement.

You're really missing the crux of fair use:

"Noncommercial, educational, critical, or transformative uses (like commentary, criticism, news reporting, parody, or research)"

How closely does writing Harry Potter fanfiction align with commentary, criticism, news reporting, parody, or research?

Fair use is more about: writing a critique about Harry Potter. Or a Weird Al style song about it. Or presenting parts of it in a paper you're writing for class.

This is all easily searchable stuff. Copyright is extremely draconian when you really look into it.