alt Hacker NewsThe state of binary compatibility on Linux and how to address it
Comments
There is no distinction between system and program libraries in Linux. We used to pretend there was one before usrmigration, but that was never good to take seriously.
The distro as packager model ensures that everything is mixed together in the filesystem and is actively hostile to external packaging. Vendoring dependencies or static linking improves compatibility by choosing known working versions, but decreases incentive and ability for downstream (or users) to upgrade those dependencies.
The libc stuff in this article is mostly glibc-specific, and you'd have fewer issues targeting musl. Mixing static linking and dlopen doesn't make much sense, as said here[1] which is an interesting thread. Even dns resolution on glibc implies dynamic linking due to nsswitch.
Solutions like Snap, Flatpak, and AppImage work to contain the problem by reusing the same abstractions internally rather than introducing anything that directly addresses the issue. We won't have a clean solution until we collectively abandon the FHS for a decentralized filesystem layout where adding an application (not just a program binary) is as easy as extracting a package into a folder and integrates with the rest of the system. I've worked on this off and on for a while, but being so opinionated makes everything an uphill battle while accepting the current reality is easy.
[1] https://musl.openwall.narkive.com/lW4KCyXd/static-linking-an...
I bought 3 linux games on DVD between 2006 and 2016. I stopped buying linux games and instead started again buying windows games. Because there is no easy way to run them. On the other hand I can just run myst1993 and most of windows games without much hustle via wine. Wine is linux only stable abi
I don't understand why they don't just statically link their binaries. First, they said this:
> Even if you managed to statically link GLIBC—or used an alternative like musl—your application would be unable to load any dynamic libraries at runtime.
But then they immediately said they actually statically link all of their deps aside from libc.
> Instead, we take a different approach: statically linking everything we can.
If they're statically linking everything other than libc, then using musl or statically linking glibc will finish the job. Unless they have some need for loading share libs at runtime which they didn't already have linked into their binary (i.e. manual dlopen), this solves the portability problem on Linux.
What am I missing (assuming I know of the security implications of statically linked binaries -- which they didn't mention as a concern)?
This is a really great article about binary compatibility!
I disagree with their idea for fixing it by splitting up glibc. I think it's a bad idea because it doesn't actually fix the problems that lead to compat breakage, and it's bad because it's harder than it seems.
They cite these compat bugs as part of their reasoning for why glibc should be split up:
- https://sourceware.org/bugzilla/show_bug.cgi?id=29456
- https://sourceware.org/bugzilla/show_bug.cgi?id=32653
- https://sourceware.org/bugzilla/show_bug.cgi?id=32786
I don't see how a single one of these would be fixed by splitting up glibc. If their proposed libdl or libthread were updated and had one of these regressions, it would cause just as much of a bug as if a monolithic libc updates with one of these regressions.
So, splitting up glibc wouldn't fix the issue.
Also, splitting up glibc would be super nasty because of how the threading, loading, and syscall parts of libc are coupled (some syscalls are implemented with deep threading awareness, like the setxid calls, threads need to know about the loader and vice-versa, and other issues).
I think the problem here is how releases are cut. In an ideal world, glibc devs would have caught all three of those bugs before shipping 2.41. Big corpos like Microsoft manage that by having a binary compatibility team that runs All The Apps on every new version of the OS. I'm guessing that glibc doesn't have (as much of) that kind of process.
Windows having multiple C libraries has its own pain points, in particular it's difficult to ship binary libraries that return allocated memory to their consumer (you either need to have the library consumer allocate the memory, which probably explains why so many Win32 APIs have this behaviour, or allow alloc/free functions to be registered). Not to mention different C libraries having their own file handle, TLS, etc state. Unsurprisingly Microsoft now ships the Universal CRT (UCRT) as part of Windows.
Isn't the compatibility promise what syscalls essentially are?
Historically, they (almost) never break and they are steadily incremented to prevent overlaps in differences of parameters.
As WASI is also implementing syscalls for WASM, I'd argue that the binary format doesn't really matter as long as it's using the same syscalls in the end.
I understand this topic is mostly focussing on glibc/muslc problems, but if you want to develop stable software, use CGo free Go binaries. They likely will run in 10 years the same way they do today.
C ABI compatibility is a mess on linux mostly because upstream maintainers don't give a damn about semantic versioning. Just take a look at the SO file headers, and how they differ from upstream "semantic" versions of the library. As long as shared objects differ in versions due to breaking changes, and as long as the C ecosystem doesn't enforce correct versioning, this won't change.
Related video about releasing games on linux, i.e. dlopen() all the things https://www.youtube.com/watch?v=MeMPCSqQ-34
> More importantly, separating the dynamic linker from the C library itself would allow multiple versions of libc to coexist, eliminating a major source of compatibility issues. This is exactly how Windows handles it, which is one of the reasons Windows maintains such strong binary compatibility. You can still run decades-old Windows software today because Microsoft doesn’t force everything to be tied to a single, ever-changing libc.
There’s no reason to believe that widespread Linux adoption would not irrevocably damage the experience. It would end up looking something like Android. The same thing happened to the internet. It was ruined by mass adoption. You can call this gatekeeping, but nothing positive has come from getting most of the world onto social media.
> GLIBC is an example of a "system library" that cannot be bundled with your application because it includes the dynamic linker itself. This linker is responsible for loading other libraries, some of which may also depend on GLIBC—but not always.
Running WordPerfect on modern Linux is done by shipping both of those components:
As an end user I often patch the glibc version incompatibility away with https://github.com/corsix/polyfill-glibc
$ ./polyfill-glibc --target-glibc=2.17 /path/to/my-program
> shipping software on Linux
That's a surprisingly hard nut to crack when containers won't work for your use case. We found https://github.com/silitics/rugix to work well in that situation.
I think the problem is that people treat Linux as OS instead of just a kernel.
You should assume every Linux distro is a different OS so when you are shipping your app for Linux you're actually shipping your app for Debian, Fedora, Ubuntu etc
Really the title should be something like “the difficulty of releasing binaries on Linux and how to work around it.” It isn’t really an atrocious state, the goal of a distro should be to get useful software into the repos. Software not distributed in an open source format doesn’t really help there.
Does Nix have any relevance here?
This article missed a critical point which is "the right way" to select a glibc ABI version: see binutils ld documentation, second part of the page related to VERSION support. This must include glibc internal symbols.
This will allow to craft ELF binaries on a modern distro which will run on "older" distros. This is critical for games and game engines. There is an significant upfront only-once work in order to select an "old" glibc ABI.
The quick and dirty alternative being having a toolchain configured to link with an "old" glibc on the side.
This article missed the -static-libstdc++ critical option for c++ applications (the c++ ABI is hell on earth), but did not miss the -static-libgcc and the dynamic loading of system interface shared libs.
Here's a thought: just distribute source code. ABI issues should be mostly fixed. Most computers can compile source code fast enough for the user not to notice and cache the results so that it's never a problem again. If you want optimised code, you can do a source to source optimisation then zip and minify the file. You could compile such a library to approximately native speeds without much user-end lag using modern JIT methods, and maybe even run LTO in a background thread so that the exectuables outdo dynamically linked ones.
Absolutely ridiculous that scrollbars are disabled for this page. The kids are too cool for such pedestrian web parts! Grow up and stop forcing your shitty UX on people.
Edit: Found it - it's black on black - even worse!
So of the 3 glibc issues they link
- one is about the format of symbol information in the actual ELF binaries which is only an issue if you are not using the standard libc functions for looking up symbols for some strange reason
- one is an issue that impacts targeting a lower version of glibc from a higher one which is a configuration that was never supported (though usually fails more loudly)
- the last one is a security policy change which is legitimately an ABI break, but mostly impacts programs that have their execstack flags set incorrectly
glibc actually goes to a fair bit of effort to be compatible with old binaries unlike most of the rest of the Linux userspace. The binaries I built for my side project back in 2015 (BlastEm 0.3.0) still work fine on modern Linux and they dynamically link against glibc. This is just a hobby project, not a piece of professional software, and a build from before this JangaFX company even existed works fine.
I find it really bizarre when people talk about Linux binary compat and then complain entirely about glibc rather than the sort of problems that the manylinux project has had to deal with. glibc is one of the few parts of userspace you can depend on. Yes, setting up your toolchain to build against an old glibc on a modern distro is a bit annoying. Sure, if you do something sufficiently weird you might find yourself outside what glibc considers part of their stable ABI. But from where I sit, it works pretty well.