Indeed. I like to say, we would like three things to be true:

1) Every day has 86400 seconds.

2) Every day is from noon (sun exactly above) to noon (sun exactly above).

3) We use SI seconds.

You can't have all three. Pick any two:

1, 2, not 3: What you describe. Day has 86400 seconds, we keep in sync with the sun, but we tweak the seconds a bit. There are different versions, like epoch time, or UT1.

1, not 2, 3: Every day has 86400 SI seconds, and we slowly go out of sync with the sun. That's TAI.

not 1, 2, 3: We use SI seconds and keep in sync with the sun +/- 1 second, but need to add/drop seconds occasionally. That's UTC.

https://en.wikipedia.org/wiki/Universal_Time

https://en.wikipedia.org/wiki/International_Atomic_Time

https://en.wikipedia.org/wiki/Coordinated_Universal_Time