My guess would be cache related. 5k probably fits in L1-L2 cache, whereas 20k might put you into L3.