It was definitely on the order of hours for large code bases - the Microsoft Excel team passed out punishment “suckers” for those who broke the build - causing 100+ people to not have a new working build to look at and test.
Linux kernel compiles in the 1990s were measured in hours, and that codebase was tiny compared to many. So, yep, builds were slow, slow enough to have an entire xkcd comic written about them.
Entire builds being slow isn't the main point though, it's iteration time from changes. I have a hard time believing people were working on a single compile a day and building an entire huge program on every iteration. That's the whole point of compilation units.