I suspect the cause-and-effect in creating the narrative is the reverse of what's in the narraitve: Frank Herbert wanted the intricate dynamics of the Guild and Spice and Mentats and exciting close-quarter combat for a more intriguing narrative. But AI and robots made those all obsolete, so he made it disappear with a handwave of "because Butlerian Jihad."

I always thought the Butlerian Jihad was the biggest plot hole in Dune, but I deeply appreciate the world and narrative it enabled.