alt Hacker NewsWhat hardware (CPU/GPU/memory) and network was used for this? What quantization for GLM 5.2? How much tuning of the split was needed?
What hardware (CPU/GPU/memory) and network was used for this? What quantization for GLM 5.2? How much tuning of the split was needed?
The lab features two Mac Studios: an Apple M3 Ultra (32 CPU cores, 80 GPU cores, 256 GB unified memory) and an Apple M1 Ultra (20 CPU cores, 48 GPU cores, 128 GB unified memory), both connected via 1Gbit Ethernet.
We use a customized Q2 quantization that preserves sensitive tensors at Q8.
To reduce compute time per layer, we are developing a custom GLM DSA Metal graph.
While we are not yet approaching MTP, we plan to port our existing MTP implementations from versions 4.7 and 5.1 to 5.2.
Since GLM's MTP acceptance rate is very high for a single predicted token, we are exploring token prediction techniques to widen the predicted tokens and utilize parallelism for verification.