Intel didn't decide not to improve. Their CPU R&D has a long pipeline and they made the wrong prediction on tech (10nm), which resulted in a decade long pipeline stall. They're just about through that now, we'll see if they've got their groove back in the next few generations.

Before the 10nm issues, they were making consistent IPC improvement every generation, and although core counts were pretty stagnant for consumers, they would add a couple cores every generation on top Xeon skus. Maybe they could have done bigger changes with more time between, but it was nice being able to expect new server CPUs every 12-18 months that were meaningfully better.