This is a terrible idea. It's just removing functionality in order to remove a button.

With this, there will suddenly be a bazillion frustrated users who want to force an exception to their timed dark/light mode cycle, but can't because this proposed system hides how to do that. And even if they look up how to do it, they have to either wait for a specific time of day to make their change, or they have to temporarily change their system settings in order to change the program/website-specific setting.

Meanwhile, people who don't care or don't understand will just leave it on the default and probably not even look at the settings page. And everyone is used to seeing "system default" as an option at this point, anyway.

Net effect: no benefit, and some users now upset.

Why change what isn't broken?