This doesn't explain why pessimistic and dystopian sci-fi has declined.

Also, Foundation takes place in a future where society is about to collapse and regress into dark ages. It's a borderline dystopian world.