The exact growing conditions for olive production aren’t common in the US, so most of the production comes from California - west of Sacramento and south along the San Joaquin river. There are a lot of barriers in bringing specialty crops to market related to know-how and contracting sale of product, so even in other areas where growth may be possible it may be infeasible.
I mean if you could make olive oil cheaper in America wouldn't someone have done that by now?
The US never lacked for smart entrepreneurs looking for a business opportunity. See wine.