Groups did sue after the Bank Secrecy Act, and the cases went all the way to the Supreme Court. The Supreme Court determined that it did not violate the constitution: https://en.wikipedia.org/wiki/Bank_Secrecy_Act

> Shortly after passage, several groups attempted to have the courts rule the law unconstitutional, claiming it violated both Fourth Amendment rights against unwarranted search and seizure, and Fifth Amendment rights of due process. Several cases were combined before the Supreme Court in California Bankers Assn. v. Shultz, 416 U.S. 21 (1974), which ruled that the Act did not violate the Constitution