alt Hacker Newsuv: Deduplicate all files in the wheel cache
Comments
Nice improvement. For me, uv is the ‘Quicklisp for Python.’ uv just let me enjoy using Python like Quicklisp just makes Common Lisp nicer to use.
I have always been a Lisp devotee, but a few years ago when I started using uv, I then started seeing Python as a language I could really enjoy using so I put effort into making my Python dev setup nearly frictionless.
uv is the backbone of any modern Python library. I’m excited to see improvements.
A 10% reduction in cache size in exchange for a 4% slowdown doesn't seem obviously worthwhile to me, especially when it comes at an increase in complexity.
> deduplication at the file level: every file is now stored under its BLAKE3 hash
Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing).
If I've got a file named:
DSC98731-b3-7b39197a22.JPG
- if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped me troubleshoot issues)
- if any other file has the same Blake3 7b39197a22 hash, it's a duplicate
- if that 7b39197a22 checksum is in my database, "things can happen".
It's really great (and I know several here independently made similar schemes) and Blake3 is an amazing hash for those kind of use.
What is the point of using packaging software that belongs to ClosedAI?
Astral people are already in the comments making pro AI coding statements. By using uv, you are literally supporting the people who want to make you unemployed for stock options.
People say uv is good because it's fast, but honestly I don't care about that. We switched to using it for distribution of our Python-based tools because it makes it very convenient to install directly from a git repository and then to subsequently update from same repository. No need to build a package.
As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can.
But it has always had two major issues:
1. No way to reproduce exact distributions for a "download" command (there is no uv equivalent of "pip download")
2. For people with a lot of different environments the cache grows significantly more than pip
I'm interested to see, at least anecdotally, if this significantly improves 2, then we can perhaps have a two layer caching strategy without the significant disk space cost.