alt Hacker News(I work on uv)
> The only advantage of uv is to have support for parallel async extraction.
This isn't true, the biggest speed ups are for the warm cases where we've already unpacked the files into the cache, as notatallshaw mentions above. We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code.
> If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv.
I'm a bit confused by these claims about the GIL? The expensive IO operations release the GIL.
> When the user edits one file, all files are modified simultaneously
This is why we default to reflinks or copy-on-write semantics when creating environments, not all file systems support it but it's becoming more common.
> Hypothetically, a simple deduplication of binary files (.dll .so) should achieve 50% of the savings without significant drawbacks
We also explored this (see https://github.com/astral-sh/uv/pull/19694) and the linked pull request has a table comparing to this strategy.
> We have optimization for empty files (0 bytes) because there is nothing to write and checksum.
Interesting, I would be very surprised if this made a significant difference? but I'll take a look.
Replies
> I'm a bit confused by these claims about the GIL? The expensive IO operations release the GIL.
FYI: The extraction of files is largely python code that doesn't release the GIL. (cf. the zipfile class from the python interpreter has large layers of abstraction with a massive overhead in python code).
Regardless, python packages are thousands of tiny files, so pip never gets to release the GIL for any meaningful duration.
If you were writing an app that only extracted large GB files, you could take advantage of some I/O operations and some zlib operations freeing the GIL for a bit. Unfortunately pip is the opposite use case, lots of tiny files.
> Interesting, I would be very surprised if this made a significant difference? but I'll take a look.
Optimizing empty files was actually quite worthwhile for pip, because about 10% of python packages are empty init files.
This might not give the same result for uv though. pip is fully linear, every single open/read/write/stat operation we removed was a direct performance gain. uv does parallel async IO, you could very well remove 10% of filesystem calls and barely affect the overall duration. :D
> We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code.
As a complete aside, uv can and does do this, but for this particular optimization I'm not sure how much absolute time it ends up saving compared to pure Python in real world resolution scenarios.
uv's total memory usage isn't that much leaner than pip's, and for parsing speed it turned out that the library pip uses, packaging, was just very unoptimized at the time uv launched. This has been significantly addressed since then:
* We did a lot of work to make version parsing twice as fast: https://iscinumpy.dev/post/packaging-faster/
* Since that blog post I made typical version parse three times faster on top of that: https://github.com/pypa/packaging/pull/1082
* Also since that blog post version filtering has gone through multiple optimizations and in some cases will be more than 30x faster e.g. https://github.com/pypa/packaging/pull/1105, https://github.com/pypa/packaging/pull/1111, https://github.com/pypa/packaging/pull/1120
At this point large dependency resolves in pip are spending very little of their time doing things in packaging, like version parsing. The main non-IO time spent in large resolves is now in the core resolver, resolvelib, which I hope to one day replace with my experimental resolver nab: https://github.com/notatallshaw/nab. Nab scales to large resolves much more efficiently than resolvelib (in fact I've cross-ported some of the algorithmic efficiency gains to uv already ;o)).