World Cups have to alternate with continental competitions (Copa America, Asian Cup, European Cup, Africa Cup of Nations) which are on similar cycles. They could technically be held every two years (and current FIFA leadership is pushing towards that), but federations and clubs are resistant (because every summer tournament places even more stress on an already-long club season, and it would likely devalue other competitions).

Clubs want their star players focused on club championships, and the star players make their real money playing on their club teams.