If you take the 4th Amendment to specifically and solely bar inconveniencing property owners without a warrant, then you are arguing that the 4th Amendment is a water sandwich. When the 4th Amendment was actually written, it was not actually possible for the investigative powers of the state to not inconvenience a property holder. But we've been able to violate people's right to privacy without them even knowing for almost a century now.

Furthermore, the last major SCOTUS case regarding this issue[0] had some very interesting dissenting opinions specifically on the question of "Does the 4th Amendment only guarantee property rights". Justice Thomas made the exact same argument you made. Justice Gorsuch took your argument and twisted it inside out. He specifically argued that because the 4th Amendment is a protection on property, the third-party doctrine should be thrown out entirely, and that you should still own your personal information even if you have to lend it to a phone company in order for them to connect you.

So yes, there are valid arguments for the other side, even in the "4A only protects property" regime.

[0] https://en.wikipedia.org/wiki/Carpenter_v._United_States