You're just conflating NFRs with spec.
Structuring your codebase like it's your spec doesn't mean you're omitting the NFRs - they need to become part of your codebase too where applicable.
eg with your first example of response times - in javaland you can use annotations on the REST resources and RPC endpoints
or your second example with specific data that may not be cached via a combination of AGENTS.md in the directory of the module providing access to it, potentially doctexts on the data constructs in question and QA agents which run pre review and raise (and resolve) the inevitable violations. Which I may add will _also happen, and so much more frequently_ when you're doing "spec driven development" via markdown files.
The difference is just down to code being testable, and markdown/yml being purely "trust me, it's like this fr fs man"
However you want to structure your spec, just structure your code like that - it works way better, because llms can handle source code so much better then markdown files.
It's not going to pass the "enterprise architect" review job application of the last decade, but your project will work, much much better and with less input required by the developer... And your development process is less let's roll the Dice and more this isn't the spec yet