"Quite often" = actually quite rare. I think you greatly underestimate the number of new novels published each year.

Your first two examples would have been covered under a 14+14 copyright period.

I do not think a 28-year copyright period would have kept Atwood from writing The Handmaiden's Tale, do you? She was a millionaire by the time that copyright expired.

I don't think looking at peak sales for outlying cases should affect copyright limits. When were peak sales for Shakespeare's Hamlet? Darwin's On the Origin of Species? Marx's Das Kapital?

The justification for US copyright is "To promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries." The problem you point out is that right can be transferred to publishers and others. Note that since 1978 it's possible for an author to terminate that transfer after 35 years, which is well after those peaks you mentioned.

What you've not mentioned is the ability for other authors to build on existing ideas. Disney famously profited by re-telling public domain stories, but will come down on you if you re-tell their stories. Speaking of fantasy, you can now write stories which take place in Oz, but make sure it doesn't have ruby slippers as that's a detail from the movie, which is still under copyright.