There was a point in time in the mid-2000s when P2P networking had briefly made running your own server attractive to end-users again. And then the iPhone came out and completely killed any hope of that becoming the norm.
The thing about smartphones is that they are both completely dependent on wireless connections to central servers in order to function and completely unsuitable to operate as servers. If you forced your phone to serve files anyway, your battery would drain quickly and the CPU would be drowning in its own waste heat. You might argue that you could still do non-server things on the phone, but practically speaking, a node that can't handle server tasks is just a leech. P2P networks work on a mutual aid basis; they require the majority of nodes be capable of shouldering traffic or sharing files in order to be a net benefit. If you add, say, tens of millions of new mobile phones to the network, the network will become unusable as any desktop machine gets DDoSed by hordes of phones asking for a babysitter.
So even in the world where IPv6 did to v4 what v4 did to NCP, we'd still ultimately end up with "my files go in the cloud", because clouds are coinventions of smartphones, in the same way that cars are coinventions of suburbs. You can't have one without the other, and once you do have both, they become so economically dominant that others get socially coerced into using them.
Nokia were experimenting with using their smartphones as web servers in the mid-2000s. You were able to write blog posts and share photos, and if you logged in to the site you could look up your contacts or download your camera roll. There's an article on this here: https://allaboutsymbian.com/features/item/Previewing_Nokias_... .
Obviously it's all very basic, but with a few years of development and polish (and probably someone besides Nokia copying the idea) I think this would be a compelling product. You could basically have the same feature set as something like iCloud but without the subscription or your files going elsewhere.