I guess you're right, but it's more of a curve, though. Once you get to any decent level of complexity, it actually helps to have a framework instead of just going all HTML+CSS. Also it helps having something standard as react (that every web developer should fundamentally understand) than doing your custom stuff if other people will be working on it in the future.
There's a lot to say about the side effects of frameworks but there's a reason why everything converges towards that.
I think it's the other way around, a framework will get you up and running quickly, but then it becomes technical dept, and if your app is complicated you will end up fighting the framework. If you write something from scratch it will take a while to reach to the abstraction level where you can work fast. But then you have a fully custom abstraction layer that is not a "one size fits all" but custom tailored for your needs.