What Was the XZ Utils Backdoor? Three Years of Trust, Caught by Half a Second
It was caught by a performance regression.
Not by a scanner, not by a code review, not by any of the tooling built for exactly this. By one person benchmarking an unrelated database who noticed that failed SSH logins felt slow, and decided that bothered him enough to look.
What was the XZ Utils backdoor?
CVE-2024-3094, disclosed 29 March 2024. A backdoor in xz Utils versions 5.6.0 and 5.6.1 — specifically in liblzma, the compression library, which an enormous amount of Linux depends on without thinking about it.
The target was sshd. Not directly: OpenSSH does not link against liblzma. But several major distributions patch sshd to integrate with systemd notification, systemd links against liblzma, and so on a patched distribution the compression library ends up inside the SSH daemon's address space. The backdoor used an IFUNC resolver to hook the RSA public key verification path.
Versions 5.6.0 and 5.6.1 reached Debian sid, Fedora Rawhide and Fedora 40 beta, openSUSE Tumbleweed, and Kali. They had not reached a stable enterprise release. That margin is measured in weeks.
The three years
Almost none of this attack was code. It was patience.
xz Utils was maintained essentially by one person, Lasse Collin, who had been open about being under strain and short of time. That is not an unusual position for a maintainer of critical infrastructure; it is close to the default.
Beginning around 2021, an account under the name Jia Tan started contributing. The contributions were real and useful. Over roughly two years the account accumulated the ordinary trust that competent, persistent contribution earns, and was eventually given release authority.
Running alongside that, other accounts appeared in the project's mailing list to complain about the slow pace of releases and press for another maintainer to be added. Read individually, each message is the kind of impatient thing open-source maintainers receive constantly. Read together, and with hindsight, they are a pressure campaign aimed at a tired volunteer.
This is the part that has no technical fix. The attack was against a social process, and the social process was working exactly as designed.
Why auditing the source would have found nothing
The malicious code was never in the git repository.
It lived in binary files that presented as compression test fixtures — exactly the kind of deliberately-corrupted sample data a compression library legitimately needs and that no reviewer reads. The assembly step was a modification to a build macro that ran only when building from the release tarball, not from a git checkout.
So the repository was clean. Anyone diffing the source on GitHub, running a static analyser over it, or reviewing the commit history, would have found nothing, because there was nothing there. The backdoor only existed in the artefact people actually download and compile.
That gap — between the source you can inspect and the tarball you actually build — is the durable finding, and it applies far beyond this project.
Half a second
Andres Freund, a PostgreSQL developer employed by Microsoft, was running micro-benchmarks. He noticed that failed SSH logins to a test machine were consuming noticeably more CPU than they should, on the order of 500 milliseconds extra, and separately that Valgrind was producing errors in sshd.
Neither observation is dramatic. Both are the kind of thing a busy person writes off. He did not, and he posted his findings to the oss-security list on 29 March 2024.
The backdoor was competent, deliberately obfuscated, and had survived multiple distribution packaging processes. It did not survive being slightly slower than expected.
Correcting the record
The comfortable reading is that open source worked — many eyes, caught in time, system functioning.
That is not really what happened. The many eyes did not catch it; the repository looked fine and the reviews passed. What caught it was one person with unusual instincts working on something else entirely, and a margin of a few weeks before it reached systems that matter.
The other half of the correction is about blame. It is tempting to file this as a failure of the xz project. The maintainer was an unpaid volunteer carrying a dependency that sits underneath a large share of the world's computing, was visibly overloaded, asked for help, and got a helper. Everything he did was reasonable. The system that left him in that position is the defect.
What is still true
The identity behind the Jia Tan account has never been established. The commit timestamps and the multi-year timeline are consistent with a well-resourced actor rather than an individual, and no more than that has been demonstrated.
Reproducible builds — making the tarball verifiably derivable from the repository — went from a niche concern to a mainstream one overnight, which is the single most useful thing to come out of it.
And the underlying condition has not changed. There are still hundreds of critical libraries with one exhausted maintainer, and the attack against them is not technical.
Further reading
- Andres Freund's original post to oss-security, 29 March 2024. Read the actual message; it is short.
- Russ Cox — Timeline of the xz open source attack. The clearest reconstruction of the social engineering.
- The Debian, Red Hat and openSUSE advisories, for what actually shipped where.
- The reproducible-builds project, for the response.
EXHIBIT 0015 — XZ UTILS
We put it on a shirt. A pocket watch opened up, its dial marked only at half a second — and where the movement should be, a set of nested archive boxes, one of them subtly wrong, with a thread running out of it and off the edge.
Most people will read it as a watch. A smaller number will count the boxes.
That's the whole idea.