> This non-battle turned out to be a powerful experience, but its power stemmed mostly from the contrast between this and my other monster encounters.
This is more correlated with modern games. Modern games, at the least non-Indie games, dumbed down the gameplay. I am not saying such games are awful per se, but they are often very simplified to the 1990s era in many ways.
Many old RPG games are quite complex; or told as a story. The old Betrayal at Krondor was kind of like a novel - unsurprisingly since Feist wrote most of the content (save various adjustments made to the gameplay itself). Yes, graphics are bad, options are too few, but storyline-wise this was my favourite RPG. Another example would be "Realms of Arcania" (in german the three DSA games). Again, graphics today are not great, and playing it in english versus german is actually worse (one of the few games where german was better than english, by the way), but the gameplay options in the second part were nice. Part 3 was a bit different, and people critisized it, but I still liked that you would explore a "real" city while still having tons of options available. Other RPGs such as Baldur's Gate 2 are a bit different - DnD itself is IMO a very bad system for RPGs (takes too long to explain now, but just look at static alignment systems - that makes zero sense) and most of it was focused on hack-and-slay for power and items, so it has the same problems. But with mods you can kind of extend the story and add more storylines, thus having more options. So BG2 is not the best example here, compared to the other two; even before that, if you remember the old Ultima series, the NPCs kind of had a regular life, worked at specific times, went to work leaving their homes (and you could then pillage that) and so forth. A lot of the "why do I want to slay the cute monster", is driven by the underlying design. These games often try to dumb down everything. I noticed this first with World of Warcraft. To me these games never were interesting, as it seems to have been deliberately dumbed down. Many of those games today are more like a movie with a bit interaction in between. That's imo not quite a game anymore. There are some exceptions though; I liked little nightmares, but this is also a simplified, mostly linear gameplay. This problem keeps on coming back again and again. For some reason modern games hate complexity. Either humans became dumber, or designers wanted to simplify things.
1. there's a trade-off between graphics/sound quality and story complexity. The better quality your voiced dialogue is - the more you have to pay for every additional line - so you tend to shorten it. Same with graphics - it's one thing to paint 8 frames of 32x32 sprites. It's another to motion-capture, model, texture, and process 100s of different versions of each character animations.
2. you're comparing unfairly (looking at the most complex examples from the past and comparing them to modern average). There were LOTS of very simplistic games in the past. You just don't think about most of them. Some genres went extinct because of how simplistic they were (see the dungeon crawlers where there was no dialogue or story whatsoever - just moving at 90 degree and hitting monsters) - it's the "old music was better" fallacy - you don't remember the old music that sucked.
If you compare most complex modern games they blow out of the water anything from the past. Let's say Baldur's Gate 3 or Dwarf Fortress or Minecraft compared to let's say Elite or Betrayal at Krondor.