> It's not that they can't outvote the farmers

Japan is a parliamentary system, and the amount of Single Member Seats in the House of Representatives needed to flip an election are primarily small towns or rural.

This is what happened in the 2009 election when the LDP lost the farmer vote.