alt Hacker NewsGit 3.0's upcoming SHA-256 default will be a costly mistake
Comments
Do I understand correctly that GitHub currently doesn't support SHA-256 repos at all, so the author faked a screenshot of what the create repository UI would look like if they choose a really stupid way to add support for SHA-256 repos, and then proclaimed it an unsolveable problem? Selecting the repository format based on the first thing pushed to it is really not a crazy idea.
The submodule problem is real, but the forge problem is entirely just "if forges implement support in a way that makes it painful it will be painful", and that's true of literally every feature that requires forge support.
One of my favorite fun facts about Fossil SCM (another source control by the devs of sqlite) is that they patched their use of SHA1 6 days after the shattered attack was published:
"Both Fossil and Git started out using only SHA1 hashes. But when the SHAttered attack against SHA1 was published on 2017-02-23, the need to migrate to a stronger hash algorithm was recognized. Fossil added the ability to use SHA3-256 as an alternative on 2017-03-01 (six days after the SHAttered attack was first published). SHA3-256 is now the default for all new repositories and check-ins in Fossil, though older check-ins that occurred prior to SHAttered can still use their original SHA1 hash. Hence, no repositories had to be rebuilt and no hyperlinks were broken."
https://fossil-scm.org/home/doc/trunk/www/hundredandone.md
To me it's so interesting watching in realtime Git is still battling with this decision and for Fossil it was just another week of development.
That whole page is fun to read. Another fun fact somewhere else in the docs is that Fossil uses a grow-only set to store commits. They came up with this scheme some years before it was formalized by CRDTs!
For reference: https://git-scm.com/docs/hash-function-transition
Notably, a few of the featured author's reservations appear to be addressed. According to the Git docs:
- Objects can be referred to by their old, SHA-1 name or their new, SHA-256 name. This means old refs in docs and comments and such remain valid. The mapping between SHA-1 representations and SHA-256 representations appears to be intentionally bijective a.k.a. 1-to-1 (assuming no hash collisions), so that it could be re-computed on demand. The constraint of bijectivity appears to be the source of some limitations, ex. no mixed repos and submodules needing to match hash algroithm, but also bijectivity has strong benefits like the following items.
- A bi-directional dictionary is maitained from SHA-1 to SHA-256 names so translations between the two don't required re-hashing objects. This table could be recomputed on demand due to the bijection between names; it's only a performance optimization.
- A local SHA-256 converted repo (including an SHA-256 converted submodule) can interoperate with an SHA-1 only remote transparently to the remote server by translating names using the lookup table.
- SHA-1 based GPG signatures will be preserved. A commit can be signed based on its SHA-1 representation, its SHA-256 representation, both, or neither. The bijection means the two types of signatures are in a sense interchangeable, or in other words the bijection between object representations implies an equivalence relation on signatures. An SHA-256 converted repo can quickly validate an SHA-1 based gpg signature using the lookup table.
Linus Torvalds in 2007:
> but the point is the SHA-1, as far as Git is concerned, isn't even a security feature. It's purely a consistency check. The security parts are elsewhere, so a lot of people assume that since Git uses SHA-1 and SHA-1 is used for cryptographically secure stuff, they think that, Okay, it's a huge security feature. It has nothing at all to do with security, it's just the best hash you can get. ... [1]
[1] https://www.youtube.com/watch?v=4XpnKHJAok8&t=56m20s
So Torvalds used SHA-1 purely because he needed a hash function with no other property than identifying content.
I think this change is more to do with politics rather than "security". Those kind of things where companies or gov, need to be certified with those super secure certificates and can't be using software that uses SHA-1. I don't have proof, but I'm not doubting it either.
This is what I saw in one of the mails. > > There are organizations where SHA-1 is blanket banned across the board - regardless of its use
And also on git 3.0 breaking changes. > > SHA-1 ... recommended against in FIPS 140-2 and similar certifications
Since SHA-1 isn't used for security in git, they should've instead moved to a non-cryptographic hash function such as MurmurHash3 and avoid all these problems, instead of moving to SHA-256 until SHA-256 is broken and need to move to the next cryptographic hash that is now incompatible with previous versions of git repositories.
I don't understand why Git is not making the SHA-1 and SHA-256 modes far more compatible with each other.
SHA1-hashed objects should be able to refer to SHA-256-hashed objects, although this seems somewhat pointless.
But SHA-256-hashed objects should also be able to refer to SHA1-hashed objects, with a major caveat: if those objects themselves are part of a collision pair, then there is a genuine problem. But this is avoidable! Suppose that Linux decided to migrate to SHA-256. The upstream project could choose a pair of dates, say January 1 2027 and March 1 2027. Up to the first date, maintainers would be welcome to submit hashes of objects that are not yet in the repo but that they think they might submit later on, and, on that date, the upstream tree would finalize the list of these objects and reference it in the repo (with a new mechanism for this purpose). Effective the second date, the repo would start publishing SHA-256 commits and would never again accept a SHA1-hashed object that was not in the repo at the cutoff date or referenced as part of the Jan 1 block.
And now it would be impossible to get a new SHA1 collision in to the repo.
The only new git features needed would be:
a) actual compatibility so that a SHA-256-hashed object could reference a SHA1-hashed object
b) a new object type that's a list of allowed SHA1 hashes (or probably a tree of them) that is itself hashed with SHA-256 and a mechanism to link to one of these from a commit
c) a policy mechanism to set a repo to only allow SHA1-hashed-objects that a reachable from a preconfigured SHA-256-hashed commit
The hashing algorithm's security properties are a security property of Git. This is because:
* we pin to commit hashes and expect this to refer to immutable content,
* commit and tag signatures are over the hash.
The Linux quote about trusting the distribution doesn't make sense to me, as Git is content-addressable and decentralised, although possibly at the time it was a reasonable position to take for kernel development, but [1] could equivalently happen for Git and the hash algorithm being non-broken is required for it to be noticed. Not having to trust the forge is a very desirable property.
From what I'd read, SHA256 in git is showing every sign of being another IPv6. In particular:
- It's implemented in a non-backwards-compatible way
- The benefits over the older model are a bit nebulous
- There's a large amount of tooling that needs to catch up, and little sign that there is movement there
This means, if you migrate your repo, every single commit message that contains text like: "please see commit <sha1>" will now be broken.
This will be a train wreck. I hope they don't release before adding compatibility modes to keep the existing sha1's around in the database.
I thought this would be a snark but it's an extremely well put together argument against the "Hashmageddon".
If you're replacing the weakness of SHA-1 just by going to another algorithm, you better be prepared to go to the next one when sha256 collisions happen, and it doesn't sound like git's design would be easy to modify for this type of crypto agility.
I do like their proposal for using signatures to establish trust and allow swapping sha256 for whatever comes next.
> The first thing that you'll notice (other than the much longer hash value) is that you can't push this code to GitHub, though that will almost certainly be fixed by the time Git 3.0 is released. In fact, that's probably the main thing currently delaying 3.0 entirely.
> But when you do want to push it to GitHub (or any host), you will need to tell them when creating the repository on the server that this is a sha256 project. Every project will now be in one bucket or the other and they cannot be mixed.
> This is immediately going to frustrate people because now they need to know what version of git they ran git init with and make sure when they go to GitHub to create the server repository, they choose the right one.
Github can fix this problem, and we have had siilar stuff in the past, when we had to switch repos. This is simple and doable, just need gradual work. I don't see the problem.
This is actually a good change. If you want to change the security assumptions of Github repository, ie make them somewhat distributed. Then the SHA-1 based commit hash is a major problem. It only costs about 10k in 2024 to find a collision to a random SHA-1 hash. While this costs essentially makes the attack infeasible for most threat models. It does limit how far you can scale this without an obvious footgun waiting for you.
This is a good change, even though there is a massive technical debt in changing such a widespread system. It is worth the effort. Should generations from now still be using SHA-1 for their Git ops? Sometime you have to do the switch, otherwise you will never progress.
For my usecase basically i needed to know that every Git commit pointed at a cannoical blob. With SHA-1 you could generate two blobs which hash to the same SHA-1 hash, while you can do the format verification which helps i could not do that in my usecase as i did not know the underlying data. To fix this i had to very ugly have two methods of referencing any Git object, a cryptographically secure SHA-256 ID and the Git ID SHA-1.
This article says if you manually enable the experimental new code today then “you can't push code to GitHub” and that old and new repos “cannot be mixed”.
The git transition document talks about keeping a bidirectional mapping between old and new hashes, and converting between the two when pushing to legacy repos:
https://git-scm.com/docs/hash-function-transition/2.55.0
I suspect that’s what gitbutler.com means when they say that, although broken today, everything “will almost certainly be fixed” when git 3.0 ships.
While I’m sure they make a good argument in the hash theory section, I’m less inclined to believe the “train wreck” part of their post.
If you upgrade your house and leave the roof off then yes, it will be a “costly mistake” the next time it rains, but if your plans say you intend to put the roof back on then I’m not sure how I am supposed to interpret a blog post warning of the perils of a roofless house.
So they’ve been talking about this for many years, planning, and finally announce when they’re going to switch the default.
So this is the right time to post that everything they’re doing is wrong? Did you engage in all the discussions about it and how best to handle it? Whether SHA-256 was the best solution?
I don’t see anywhere that it talks about alternate proposals or why they might have been better. Why the particular suggestions here were rejected.
This seems like a bunch of Monday morning quarterbacking.
Yes every repo is either one or the other but you fix that by rehashing the entire repo. Everyone can do this independently. It's entirely possible to maintain to identical repos in SHA1 and SHA256 mode but for the most part I suspect once updated people will simply pull down the new repo and use git 3.0 as a required version.
As migrations go, it's reading as simple to me. You'll just have to backpoint the commit signatures. I must assume there's a backwards compatible reference for them in git 3, right?
Or drop them and reference the old structure in a dire pinch.
I find it disappointing how few people are addressing the proposed "Independent Tree Hash Headers" solution. It's probably the most interesting part of the article, but it's getting the least attention.
I came in expecting to disagree strongly with the article, but ended up agreeing more than I didn't (though I still don't 100% agree, as collisions are still an issue for mirrors). I find the concept of multiple hashes per commit quite interesting. It would allow mixing hashes in one repo, wouldn't break submodules, and tooling could be used to reject commits without any secure hashes for a gradual transition (like enforcing signed commits/tags).
>it will be an incomprehensibly expensive and ultimately valueless and avoidable global nightmare.
thought "costly" in the title and "incomprehensibly expensive" in the subheader meant this piece would discuss how much less performant sha-256 is on modern machines, but didn't see anything. isn't there hardware acceleration? how much worse is it?
I agree with the main thrust of the article, that we should instead trust the transport mechanism rather than the cryptographic properties of SHA1, but because of this I don’t really think switching the default will be a costly mistake. I rarely use full SHA1 hashes right now; I only use the truncated version and I don’t think any user cares about the length of the full hash. As for compatibility with forges, it’s just a small UX problem that should be solvable: don’t let the user choose the format when a repo is created; instead choose it when the first push happens.
I don't agree that SHA-1 is much longer feasible for git.
But I also don't think that switching to SHA-256 must be painful. A git2->git3 converted repo could just store all the past hashes, so existing links don't break.
Meh. Git 3 can still init with the old sha1 hash.
Also GitHub should auto-detect the format on the very first “git push” and handle it then. If they actually require config to be set appropriately during the initial “new repository” dialog that’s just bad ux.
As for this being a looming disaster for industry… it’s inconvenient. And lots of things will break. But we will survive and come out the other side I’m certain. We converted from the Julian calendar to the Gregorian calendar 500 years ago. Surely we can handle this.
Core of the issue seems like github UX issues that they can solve
I'll probably switch to git-evtag then, and keep the old SHA-1 then. Same as Google.
the sha1→sha256 flip is gonna break every script that assumes 40-char hashes. my own deploy glue does that lol. not looking forward to the grep day
I positively don't care about the collision issue.
If you need to certify the authenticity of some code, and you've decided that a Git hash of any kind is going to be your certificate, you have a problem between keyboard and chair which is not fixable by stronger hashes in Git.
I don't want instability and churn in tooling.
The post's argument that hash collisions are irrelevant in practice is not convincing at all. Basically they amount to:
1. Collisions aren't as bad as preimage attacks
2. Even if you made a file-with-malicious-hash, how would you get people to pull it?
3. Other attacks are a bigger problem (social engineering)
(2) is laughable in a world with github. It's common for unknown people to submit pull requests to code bases, and for those changes to be reviewed and merged. For example, as part of reviewing pull requests, I have `git fetch`'d proposed changes to my local machine to check behavior on some additional test cases. "If you fetch it you're fucked" is unacceptable as a security boundary.
(1) and (3) are just tu-quoque arguments about other attacks being worse. The relevant question isn't how bad other attacks are, it's how bad this attack is.
The fundamental problem with collisions is that software often assumes they can't happen (or is not tested against them). Thus collisions can trigger bugs, or otherwise cause surprising behavior. For example, webkit figured the colliding PDFs demonstrating a sha1 collision would be excellent for unit tests, so they merged the PDFs into their SVN repo... which completely fucked it [1]. I don't know the exact internals of git so I can't comment on how you would get surprising things to happen, but "oops the file you merged was different than the file you reviewed" and "oops the repository got corrupted" seem entirely plausible.
[1]: https://www.reddit.com/r/programming/comments/5vyhy2/webkit_...
Ugh, I didn't know that SHA-1 submodules wouldn't be supported in SHA-256 repos. That changes the transition from painless to a major dumpster fire. Having to maintain converted forks, and use different hashes from upstream is going to be a mess.
Can someone more cyber-pilled than me explain what the actual risk with Git hashes being susceptible to collision attacks is? Obviously accidental collisions are problematic, but to my understanding the probability of that is still approximately zero.
Best I can tell, all a forced collision would do is let someone who already has control of a repo modify the history in a far from plausibly deniable way. Which in practical terms, they already could do simply by replacing the whole thing, because who's out here using git hashes as a security tool? Every pinning I've ever seen has been to tags (which can be modified at will), or hashes of the actual payload (which doesn't need to be the same as what git uses).
> We can go through years of this SHA-1 to SHA-256 migration and then quantum computers break 256 and we're back in the same stupid boat again.
SHA-256 is considered quantum safe by the NIST and is left out of PQC migration guidance entirely.
Oh, agreed this sounds like a terrible migration path and shouldn't really be needed in the first place.
What I'm missing in the article is whether any Git server accepts replacing a SHA-1 identified object it already has. If it doesn't, then the distribution trust discussed holds, and keeping SHA-1 seems fine. Adding additional signatures seems fine for those who need transitive trust.
I suspect everyone renaming their branch from master to main caused more unnecessary breakages and toil than this ever will.
Make git init use SHA-256 if git is invoked as git3, SHA1 if invoked as git2.
Plain git init could fail with a diagnostic: informing to use one of the two aliases or an option.
What people don't want is making git repos SHA-256 by accident and finding out later that they made repos not compatible with older git.
schacon: Really like the "Independent Tree Hash Headers" idea. How difficult would this be to get this functionality into git? Would it cause any breaking changes with older versions? Have you discussed this with any git devs to see if they are open to adding it?
I think the only good argument here is that sha is maybe not a security control for git. I think every other argument in this article is incorrect
a) it's relatively fast and impossible in a practical sense for two different files to accidentally hash to the same value.
That is silly. We are not worried about accidentally triggering. We are worried about intentional triggers.
I dont know why people always bring this up for hashing. In any other context it would be considered silly. If someone said, the chance of triggering a buffer overflow by accident is low, we would call that silly as we aren't worried about accidental triggers.
b) second pre-image vs collision. In a world of open source where we accept commits from randoms on the internet, i think collisions are just as relavent as second pre-image.
I run a small git/jj forge and for us it's already painful dealing with this. Can't imagine how GitHub is going to handle it.
The xz backdoor is basically his point in practice — that was a maintainer-trust compromise, not a hash collision.
My prediction: 20 years from now everyone will still use SHA-1 git. That will be simply easier.
It seems people are missing the point: it's not even the submodule incompatibility that's going to become an issue majorly (like python2 --> python3 but worse), the main issue is the loss of traceability for repos that changes in place (which I assume many will do). Imagine what will happen to these:
- SLSA and Provenance or SBOM data in the supply chain security that uses commit hash. All the previous images are now pointing to a non-existing commit
- All the documentation and tooling as the article calls out
- All your traceability links from your project tool to your git repo, they will lose all the past data as it will be dead links
So I hope there IS NOT a migration path for in-place replacement!
I'm going to completely ignore the first part of this post because I'm not interested in arguing about how severe the issues with SHA-1 are. I think it's accepted that there are flaws.
So given that, I'm more interested in the arguments for why migrating to SHA-256 is problematic.
The biggest issue I see, after skimming over it, is the submodule breakage for new projects trying to link to old projects. This seems solvable frankly, but is the only serious issue I see. Everything else will be worked out as software is updated IMO.
> not really practical to exploit in any demonstrated way
like gitc0ffee?
I feel like if it takes this much ink to explain why using an insecure primitive is actually safe, you should just fix it.
The arguments make sense, but how ironclad are they? How confident are you that some clever black hat won’t figure out a way to take advantage of it?
This is one of the most widely used programs in the world. Let’s close the hole.
The first reason the author lists for why this will be bad is only an "issue" on Git hosts that don't allow repo creation on push (which is brain dead of GitHub). Any other host, you push your new repo, and it will see the hashing algorithm, and receive the contents accordingly.
Submodules is a legitimate argument against this, though I don't know how widely this feature is actually used, and similar to the arguments in favor of switching the default branch from master to main, this is simply a setting which can be changed.
I do like the idea of commits having both hashes, and am surprised that idea has not been explored further.
Generally though, I think the author's strongest argument is simply that the change isn't strictly "needed", and all the other issues presented aren't the strongest arguments against change.
I'm honestly still shocked any of this happened.
Prior to SHA1 we had MD5, a decade earlier. MD5 collision attacks had already been widely documented and known. It was the most obvious thing on Earth that this would happen to SHA1 too. Apparently, Linus never realized there was a need for cryptographic security and that the hash was purely internal.
Here's what I honestly think was a factor. I think C programmers fell in love with the implementation that you could throw around a fixed hash record on the stack. It's incredibly efficient. But it's an efficiency that doesn't really matter because as soon as you read from or write to a disk or a network or even memory, any cost saving is completely gone.
More than a decade ago, some people wrote a Java implementation of git (jgit?) and despite all their optimizations, it was (IIRC) only half as fast as C git. It is of course because Java at the time had no concept of stack values for non-primitive types so couldn't compete. Personally, I was impressed: only half the speed? That's pretty good.
For something that's only 20 years old, the Git SHA1 assumption is some of the worst technical debt we have in the modern era.
Here's another thought: when people make a lot of these programs, they often make the mistake of not separating the program version and the network protocol (or just the external API). So you end up with brittle client-server implementations where you have to upgrade both the client and the server at the same time because they lack a network abstraction.
The other end of the spectrum is video streaming where you have codex, container formats, transport protocols and so on.
What a mess.
A lot of replies here seem to be asserting that this "isn't that hard" without addressing the thing that makes it most hard: submodule compatibility and the breadth of tooling
So many people coping.
> I pull it from there because I trust that GitHub has its authentication game together enough that it's unlikely that anyone malicious pushed something there without the maintainer's knowledge.
Hahahahaha, that's some level of delusion
This seems like Y2K fud.
The alternative to making sha256 the default is to leave sha1 the default. Nobody changes to sha256. sha1 is broken in 10 years. Suddenly everyone has to switch all at once on the same day because it is a critical security issue, but github never implemented sha256 because they didn't have to. This would be a major problem.
This is very very easy to fix if you run into it.
1. Adopt git 3.0 if you can with sha256.
2. If you can't use sha256, set the config to put things back to sha1. Wherever you need to do this you probably already set dozens of ENV vars or settings, just add a new one.
Or write a 15 page analysis about how the above is so hard people will probably just find it catastrophic to even think about.
Thanks for writing this. I'd only been loosely following it and I hadn't realised how bad this is going to be. I have repos with tens of submodules and it's going to be a nightmare if any of them switch to sha256 in place. Not to mention I won't be able to use any new projects unless I rebuild my repo and all the submodules therein.
I thought the master to main thing was bad enough but this is going to suck. And just like the master rename it achieves basically nothing.
What is it about these projects that attracts people who just want to change things for the sake of it? Real engineering means coming up with a solution for backwards compatibility. This is just irresponsible and, frankly, a fuck you to everyone who will be affected by this.
This article is full of mistakes and misleading claims:
1) It's claiming SHA1 insecurity is theoretical, while SHAttered from 2017 was specifically a pratical proof of concept. The only reason Git wasn't affected, is because they didn't bother bruteforcing a git-blob prefix.
2) It's claiming collision attacks don't matter, only second-preimage attacks do. This is incorrect, collision attacks are enough for code-smuggling problems, when two repositories are on the same git commit (verified by the full commit hash), yet contain different code in their git checkout.
3) The Linus quote "The real security is in distribution" is arguing that "git's content-addressed system should not be used to address content". It's arguing that, in case of curl|sh, you shouldn't use a sha256sum-gate to pin the content to something you've reviewed, you should instead ensure curl is fetching from an https server.