If it was a competition in who did more harm the distinction may be relevant but in terms of saving the environment and turning back the ecosystem back to a sustainable one, the distinction between animal based or non-animal based is mostly irrelevant. Modern agriculture are not sustainable for the environment. The fossil fuels that are pulled from the earth and put on fields are not sustainable, and the amount of run off that goes into the water are destroying ecosystems with no time table if they ever can recover. When different species goes extinct they stay extinct, and the distinction that "well, its not as harmful as animal-based agriculture" will not bring them back. The Baltic Sea an loud warning signal of what happens if we continue to go down this path of modern agriculture.

One of the few areas of sustainable farming is aquaculture like shellfish and seaweed, which could actually be used to reduce the negative effects caused by modern farming. If there were a competition in least amount of harm, those would likely be the winners.

Fields of corn or soybeans will still exist without animal-based agriculture, especially with current demand for biofuels. As long as the land can be farmed to generate revenue, people will farm it. Artificial fertilizers is the primary enabler of this.