It's not the craziest idea. A tractor is basically just a big hydraulic pump driving a bunch of linear and rotary actuators (commonly called "motors" and "cylinders"). Especially if it's got a hydrostatic transmission. If you design it in such a way that it's relatively easy to adapt different clutches and bell housings, maybe with a little driveshaft and u-joint between the clutch and the pump, you could theoretically accomplish something like this.

However one major sticking point is that (often.. maybe always?) the engine block casting is actually a structural component of the tractor "frame". Unlike e.g. a truck that has its driveline mounted between frame rails, a tractor's "frame" is its driveline . So this might add quite a bit of complexity and cost.