While I don't have the exact details of transport and whatnot, much of that rice came from the Carolinas https://inlandrice.charlestoncounty.org/economics.html

Generally speaking, rice is a lot higher yield per acre, and also, the east coast doesn't have particularly good wheat growing.