xz backdoor: 5 Lessons

CVE-2024–3094 — also known as xz-backdoor or xz-trojan — is the most concerning software supply chain attack to date.
Interlynk
April 13, 2024

CVE-2024–3094 — also known as xz-backdoor or xz-trojan — is the most concerning software supply chain attack to date.

A developer identified as Jia Tan (Github JiaT75) used social engineering to enter the open-source Tukaani project and earned the community’s trust over the next few years.

Jia eventually earned the maintainer status for the XZ-Utils project.

Finally, Jia used his status to deploy clever obfuscation in source code and binary files to create a backdoor in XZ-Utils versions 5.6.0 and 5.6.1. If it had remained undiscovered until integration with key distributions, millions of Debian and Red Hat releases could have been affected.

A well-written timeline of events is published here.

1. xz is the most advanced software supply chain attack

Software supply chain attacks involve ingesting malicious code or components at a layer deeper than the application itself. This attack on the “supply chain” of software build helps gain an advantage in distribution as more than one downstream project might use the same supply chain.

In the past, compromises have targeted widely used applications (e.g., MOVEit — CVE-2023–34362), a shared component (e.g., log4j), or a part of the built environment (e.g., SolarWinds, Codecov).

On the other hand, the xz-backdoor starts by first compromising the “maintainer” list of the project. This was done by creating a specific identity for the purpose — JiaT75 (no record of this user’s contributions before 2021), earning the project owner’s trust over time, and finally applying social engineering to pressure the project owner into accepting that user as maintainer.

Once in a trusted role, it is only human that code reviews have trust bias and can cause even simple obfuscation to go a long way.

The technical side of opening the backdoor is also clever. It includes using broken binary files to evade code review, loading those files as scripts using simple but divided command sequences, and splitting the total backdoor entry over many different changes. In any sizeable, fast-moving, or low-resourced project, these could be missed during code reviews by a plurality of teams—not to mention when the submitter is already a ‘trusted’ maintainer.

Combining the above, we cannot think of any other software supply chain with as many layers of preparedness before the exploit.

2. Could be happening now or soon again

Many parts of the xz-backdoor were being implemented because of the maintainer's trusted nature, which had been socially engineered over the years.

With over 50 million active open-source projects, the likelihood remains high that ‘trusted’ maintainers are waiting for the right time to introduce a malicious change or have already introduced it but remain undetected.

This differs from log4shell (CVE-2021–44228) or equivalent vulnerability, where the developers did not imagine a specific exploit in the first implementation. Therefore, post-discovery, researchers can still quickly investigate and surface specific vulnerabilities.

In contrast, a ‘trusted’ maintainer introducing a known exploit would be deliberate in their effort and, therefore, would attempt to obfuscate the changes whenever possible. For this reason, Tukaani’s builder, Lasse Collin, first reaction was —

I would be suspicious of even older versions of xz until proven otherwise.

3. None of the known security tools would catch xz

While Interlynk is a security tool vendor, we haven’t seen clear evidence of why any specific tool or technology would catch such a change beforehand.

OpenSSF first suggested that the OpenSSF Scorecard would score low for xz utils projects, which is a true statement.

However, specific checks would always be ignored in favor of projects' reputation for incredibly useful and reliable compression utility. I can’t imagine what SAST/DAST patterns will find problematic with the approach, especially considering that the offending script is pre-converted to a blob. A runtime behavior change might generate some noise, but being part of `sshd` will likely not flag suspicion in every situation.

We understand the security vendor community's need to reassert their roles in this conversation, but we believe the real conversation needs to happen within open-source communities and can be supported by tools as needed.

The root of this challenge is that knowing if someone is a Manchurian Candidate is a complex problem because the evidence remains inside their head.

4. SBOM will not detect a backdoor

Interlynk secures the software supply chain by automating SBOM flow. In the xz-backdoor, an SBOM would help identify when and what changed through the supply chain, what version each component is at, and the integrity of components (including test data).

In addition, Interlynk creates a risk score based on the OpenSSF Scorecard itself, with the package's age and utilization, the number of maintainers, and a few other signals. The specific version would have started low (5.4) and grown in value over time (6.8 by the end of May).

However, this information would trigger policy violations if configured around risk but will likely be ignored by product security teams, given the project’s earned trust.

So, we do not claim an existing SBOM program (or any other security tool) would have caught integrating xz-backdoor.

As suggested, SBOM is not a silver bullet for all software supply chain risks. However, it plays an essential role in ensuring integrity and organizing post-remediation efforts.

5. But SBOM is a solution to remediation

Imagine that this xz-backdoor was discovered in late June instead of late March. By then, it would likely have entered multiple Linux distributions and been installed on millions of servers.

All product security and DevOps know the feeling of what comes next — answering questions like:

Are we infected?

What products and versions are infected?

How do we patch them?

How do we ask our suppliers if they are infected?

What about our vendors?

How do we prove to our compliance teams that this is being tracked?

This isn’t the first time this exercise has happened, and it won’t be the last.

However, with SBOM, all the tracking could have been done in one interface.

All the assets with SBOM are mapped to the software's specific version with the components' specific version.

An upgrade plan would upgrade SBOM with it and show the component version moving to the fixed version.

An accompanying VEX would enable upstream and downstream partners to communicate this in real-time.

This isn’t going to stop, but SBOM is the building block for coordinating efforts to mitigate this.

We are here to support a healthy, safe, and incredible open-source community around us.

Reach out to us at hello@interlynk.io for a chat.