Total energy input to agriculture in the US is less than 2% of total energy consumption. So "huge" there has to be taken in context.

All the energy inputs to agriculture could be replaced with non-fossil inputs. Fertilizer in particular needs hydrogen to make ammonia, but that can be produced from non-fossil sources.