You don’t necessarily need it to learn during the game, it would be enough for it to learn between games. If you’ve played the game long enough there are behaviors you can exploit that wouldn’t work against a human player. They iterated on the AI in Beyond the Sword and fixed some of the more abusable mechanics in Civilization V (e.g. by introducing diplomatic penalties when you camp units next to an AI’s borders), but it’s just inevitable that once you’ve played a game long enough you will find these kinds of exploitable patterns.

The customization available in IV makes it basically infinitely replayable, but the AI makes the trajectory of each game too predictable if you understand the mechanics well enough.

Lots of old strategy games have been revived by introducing new factions that change the game’s meta; imagine if this process was automated by training the AI on recorded games from the entire playerbase, or on games recorded locally to adapt to the user’s unique style of play.