It's worse than that even: there are delays after every _bit_ transferred, due to various design decisions that were made along the way, which is why the C64's disk drive is slower than the previous computer (the VIC-20), which is slower than the one before that (the PET).

I wrote about the decisions and the resulting delays for one of those 100-post Threadapalooza projects in 2024, compiled here for easier reading: https://imrannazar.com/articles/commodore-1541