In the case of the legislative branch, the Constitution actually says pretty much exactly that. See Exceptions Clause of Article III, Section 2