Binary Dependencies: Identifying the Hidden Packages We All Depend On

Its possible to avoid all of those binaries (including the Linux kernel) and build from source instead.

https://bootstrappable.org/ https://lwn.net/Articles/983340/ https://github.com/fosslinux/live-bootstrap https://stagex.tools/

Seth Larson gave a talk on this (with a focus on Python as well) at PyCon US last year[1] as well.

It's a non-trivial issue, in terms of balancing conflicting interests: Python (like most interpreted languages) has a story for integrating native libraries, but that story is not particularly user friendly (in terms of users, Python developers, etc. not having the domain expertise to debug failing native builds). So these ecosystems tend to develop bespoke mechanisms for stashing native binaries inside package distributions, turning a build reliability problem into an introspection problem.

[1]: https://www.youtube.com/watch?v=x9K3xPmi_tg

This is one of the reasons I like having a nix flake in all of my projects that defines a dev environment, and integration with direnv to activate it. The flake lockfile, combined with the language-specific lockfile, gives a mostly complete picture of everything needed to build/deploy/develop the package.

> In almost all ecosystems, it is difficult to keep track of binary dependencies. When you depend on a package’s source code, this is normally recorded in your manifest file — pyproject.toml, package.json and so on. However, when you depend on a package’s precompiled binaries, this information is usually not recorded anywhere. This means that the binary dependency relationship between your project and whatever you’re depending on is hidden — so we can say that you have a phantom binary dependency.

I know it comes up every time... but nix does kinda exist to solve this problem. At least in pure mode.

Personally I like using Debian packages to keep track of source and binary dependencies.