logoalt Hacker News

Argonautlabstoday at 8:08 PM5 repliesview on HN

Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their released MXFP4 precision; the resident attention trunk is int8, which upstream labels non-weight-exact, so I don't claim bit-exactness against BF16 — I claim token-identical output against my own reference on the prompt of record, checked on every promotion. Numbers, with the unflattering ones in the same paragraph as the good ones: 1.00 tok/s steady over a 512-token completion, 1.13 over 128 tokens, and 0.96 median on the 17-token benchmark from the upstream repo's issue #15 against the 0.684 posted there. Time to first token on a 512-token prompt is about 6.3 minutes — prefill is currently read-amplified 6.2x, which is the biggest open problem in the repo and is described in the results directory. What I think is actually interesting isn't the number, it's that four of the gains came from defects in the read path that instrumentation found and I would never have guessed. The instruments are in a second repo, ARGODRIVE — a 10 ms per-device read monitor, a per-read barrier trace that records which drive served each expert and which one landed last in every pass, and a config assertion harness that refuses to record a benchmark unless the setting under test actually fired. They're deltafin-specific today. The four findings: • A constant capped the reader threads at 16 and bounded both the demand and prefetch pools with the same value. Separating them was +14%; demand queueing went from 70% of blocked time to 7.5%. • Splitting each hot expert's read across two replicas on two devices was +10% — after the same knob had measured negative six times on layouts where every expert had one home and there was nothing to split against. • The prefetch path had no balancer at all: it walked a fixed directory order and took the first hit, so on any replicated layout it dumped everything on one enclosure. Giving it least-expected-completion dispatch with in-flight counters shared with the demand path was +11% and turned every replicated layout I had previously measured as a loss into a win. • A recorded "law" that a given draft depth was worse turned out to have been measured against a drafter that no longer existed. Re-testing it was +8%. There's also a drive-count ladder in the repo — same layout, one to four drives: 57% / 78% / 92% / 100% of the four-drive decode rate. And a catalogue of about a thousand timed runs of things that did not work, with the numbers: RAM expert caches from 8 to 40 GB (-4% to -48%), striping a single copy (-7 to -25%), two drives sharing one Thunderbolt link (-11%), streaming the attention trunk from SSD (-60%), Metal's file-loading API (-19 to -22%). That catalogue is the part I expect to be most useful to other people. The engine is a fork of gavamedia/deltafin, which is MIT and did the hard part; I've told the author about all of this and the upstream-relevant fixes are going back as PRs. Two things I'd genuinely like help with: whether anyone has done expert-major prefill scheduling on an MoE (read each expert once per layer and run its kernel over all rows routed to it — it should take prefill from 6.2x amplification to about 1x), and whether the drive ladder reproduces on other hardware.


Replies

pavlovtoday at 8:47 PM

This response is so dense with numbers and special characters that it's probably about 1000 tokens. So at 1 token/s, it takes almost 17 minutes to generate this on the MacBook Pro.

show 2 replies
anigbrowltoday at 10:06 PM

So we meet again, wall of text. I thought I had killed you in Istanbul.

copperxtoday at 9:03 PM

The hyphenated terms get worse and worse as you keep reading. Just kill me now.

sampullmantoday at 8:46 PM

This is difficult to read, maybe just link to a gist?

show 2 replies
NooneAtAll3today at 10:06 PM

if anyone else is lost at how big/small 512 tokens are - this wall of text is 843 tokens, according to https://token-calculator.net

so... enough for 1 question + 1 long answer?