How much time is spent interfacing between userland and the kernel? Can you try to get it to run as a kernel module? :)

Also in case your CPU is old enough, did you try disabling CPU bug mitigations?