There are three issues that arise

- structural: There are height and size limits on things you can build without a permit

- trespassing: flinging objects onto others people property is illegal

- endangerment: any activity that can endanger other people is illegal

All of these three depend on how rural you live I guess. So in a suburban neighborhood I would assume they get involved very quickly.