This link has more on the architecture: https://cloud.google.com/blog/products/compute/tpu-8t-and-tp...