alt Hacker NewsMy first version used plain `mmap`. On the 8 GB M2, a cold 3.36 MB expert took 10 ms with mmap and 2.8 ms with `pread`. The full simulation was 0.50tok/s for `mmap` vs 4 tok/s for `pread`
With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work
And common weights still use mmap for simplicity
So, I believe llama.cpp might run it under 2gb, but I assume it will be slower
Replies
nijave • today at 5:13 PM
Any idea if madvise helps? Admittedly I have very limited experience and only on Linux
➕ show 2 replies
a-dub • today at 4:47 PM
for a given expert, do you have a sense for what the spatiotemporal access pattern looks like?
➕ show 2 replies
hm. in linux you have MAP_POPULATE which forces a prefetch where macos relies on page faults and lazy loading. if MADV_WILLNEED doesn't help, maybe readv to vector read directly or mmap+writev (write to a dummy fd, with an iovec for each page, hopefully resulting in a one-syscall-big-pagefault for mmap.) maybe also experiment with a loop that just reads one byte from each relevant page after mmap but before real computation?