Defendants don't invoke that because in most states and federally, they build the case against you slowly over a long period of time before arrest, then stall as long as possible for discovery, then when they finally fulfill discovery they overwhelm you with a bunch of useless stuff so that it takes forever to get the useful information. Invoking right to speedy trial means the prosecution gets a very strong advantage to the defense.
I don't think it's a sensible interpretation of the constitution given the massive asymmetry of the situation. The state should be obligated without exception to either provide for a speedy trial or to release the defendant while the state figures its shit out. It should not be a right that can be waived. Meanwhile a defendant who's been arrested should generally be given as much time as he'd like to put together his defense.