I've been running local image models on an older laptop recently, and memory behavior surprised me more than raw inference time.
One experiment briefly pushed private memory past 18 GB before I changed the allocation behavior. After terminating the worker/process, it dropped dramatically.
It made me realize how different "the model fits in memory" is from "the whole inference pipeline behaves well in memory."
alt Hacker News
I've been running local image models on an older laptop recently, and memory behavior surprised me more than raw inference time. One experiment briefly pushed private memory past 18 GB before I changed the allocation behavior. After terminating the worker/process, it dropped dramatically. It made me realize how different "the model fits in memory" is from "the whole inference pipeline behaves well in memory."