I think they just swapped out LuaJIT's modified built-it dlmalloc[1] with some standard allocator. Then just set some turning values of the allocator to make to more eager to return pages with no allocations left to the OS. LuaJIT has always had pluggable allocator system you can set at state construction time[2], it did have a restriction you could only use the built-it allocator for 64 bit builds if you don't use the GC64 build option, but thats been default enabled for a while now.

[1]: https://github.com/LuaJIT/LuaJIT/blob/v2.1/src/lj_alloc.c

[2]: https://github.com/LuaJIT/LuaJIT/blob/707c12bf00dafdfd3899b1...