How to Secure Open Source Software: The Dilemma of the XZ Utils Backdoor

Introduction

There are few things that one might expect to stumble upon by chance–a lost dollar in your couch cushions, that favorite sweater that somehow disappeared last year at the bottom of your closet, a song from high school that you haven’t heard in years. However, it’s not often that one stumbles upon a backdoor that has the potential to result in a debilitating software supply chain attack. 

This is exactly what happened to Microsoft developer and engineer Andres Freund a few weeks ago. The tale of the XZ Utils backdoor is the latest stunning example of the dire need to secure our cyberspace, starting with open source software (OSS). 

Zooming In–What Happened? 

On Friday, February 29th, Freund discovered a backdoor in XZ Utils, an open source toolset that compresses and decompresses data during different operations that is heavily used across the Linux ecosystem, when troubleshooting the Debian Linux beta. With the right encryption key, an actor could exploit the backdoor and execute potentially malicious code. 

This backdoor was the fruit of a multi-year operation, with a GitHub user under the name of JiaT75, otherwise known as Jia Tan, acting as the main driver. The person or group behind Jia Tan worked for two years to build the credibility needed to implement an advanced backdoor into versions of XZ Utils, providing hours of largely benign volunteer coding to win the trust of the community behind the tool. This endeavor was aided by sock puppet accounts, accounts under fake names that had no history prior to their actions in this operation, that pressured the original author and maintainer of XZ Utils to bring in another maintainer–namely Jia Tan. Following this pressure and in February 2024, JiaT75 issued commits, meaning the account submitted and saved changes to the project’s source code, to versions 5.6.0 and 5.6.1 of XZ Utils that implemented the backdoor. Then, JiaT75 and different sock puppet accounts urged maintainers of Linux distributions to update to these new backdoored versions of XZ Utils, including a few of the largest Linux distributions like Debian, Red Hat, and Ubuntu that are ubiquitous in the software ecosystem at large. 

Fortunately, Freund found this backdoor before versions 5.6.0 or 5.6.1 were merged into any production Linux distributions. Nevertheless, this vulnerability was inches from causing widespread damage. A major global cybersecurity crisis–potentially taking the form of a wave of ransomware events or as an unseen Achilles heel allowing state actors full access to a wide swath of computers around the world until detected–was only averted by one engineer noticing an uptick in a normal task’s use of computing resources. 

For more details on this vulnerability, check out articles from Wired, New York Times, Akamai, and Evan Boehs. 

Zooming Out–What Does This Mean for Cybersecurity? 

Can OSS be Trusted? 

While some may use this vulnerability to support their argument that OSS is inherently insecure and should not be used in proprietary systems, this backdoor means no such thing. In fact, this backdoor highlights one of open source’s strengths: transparency. Some experts argue that this vulnerability was only discovered before causing widespread damage because it was open source. Open source platforms, such as GitHub, are built to log and attribute all activity, providing a richly detailed history of any project’s evolution. Freund was able to analyze the code and find the backdoor because of its open source nature. Similarly, OSS sleuths were able to track down the source of the backdoor and dissect all of Jia Tan’s history on GitHub because that information was public. In addition to its transparency, OSS is cost-efficient, creates lower barriers of entry, and allows developers to focus on innovation in lieu of rebuilding the wheel.  

The 2024 Open Source Security and Risk Analysis Report found that 96% of the studied codebases contained open source code. Nearly all of our systems, databases, and tools today are built on OSS or use an open source tool to accomplish part of its work. The expense that would be involved with pivoting from this foundational OSS, coupled with the sudden expense of maintaining systems and tools without the work of devoted volunteer open source coders, makes the question of its inherent security more or less moot. Instead, the XZ Utils vulnerability should be yet another push for us to prioritize making OSS more secure.  

How Do We Make OSS More Secure? 

There are already many initiatives endeavoring to improve the security of OSS. Through the Cybersecurity and Infrastructure Security Agency’s Secure by Design program, the federal agency is working with the open source software community to ensure that security is incorporated from the onset of software development, including through promoting the adoption of memory safe coding–coding with languages devoid of memory safety vulnerabilities–where possible. 

The XZ Utils backdoor is also one example that illustrates an increased need for checks and balances in OSS. Though time consuming, manual or automatic reviews of all submitted changes to a project could be critical to finding additions of malicious code. While there are certainly additional checks and balances that the OSS community can implement, it is important to note that OSS developers and maintainers are uncompensated and already overworked.  

Consequently, the private companies who benefit from these volunteers’ work will be vital in improving the security of OSS. Strategic Objective 3.3 of the National Cybersecurity Strategy stipulates the need for a liability framework that shifts “the burden for cybersecurity … onto the organizations that are most capable and best-positioned to reduce risks for all of us.” This liability framework will hopefully incentivize proprietary software enterprises to take more responsibility for the security of the free open source code that is critical to their products.  

This responsibility should take multiple forms. To secure our cyberspace, these enterprises need to spend more resources vetting OSS–either reviewing their open source code or setting higher standards for what open source projects they engage with. Additionally, just as federal agencies, and soon federal contractors, are required to provide software bills of materials (SBOM), enterprises should similarly be expected to know exactly what software they use. With SBOMs as an industry standard, proprietary software developers will be able to more nimbly mitigate OSS risks. 

However, it is not enough for an enterprise to just manage their own open source code. As Secure by Design encourages and a liability framework will ultimately incentivize, proprietary software developers using open source code need to actively support their open source communities through contributing back code, developing award programs, or employing maintainers. This support is vital to fostering a healthy software ecosystem and ultimately making OSS and proprietary software both more secure. 

Conclusion   

The XZ Utils backdoor is a heroic tale of averting a major crisis–but also a cautionary tale. The sophistication and patience that went into building this backdoor gave it an incredibly high chance of not just succeeding but also of staying successful, keeping a backdoor open into computer systems around the world. As attackers evolve in their approach, we must also evolve our approach to security. Unfortunately, there is no quick or easy solution to securing OSS. In pursuit of a more secure cyberspace, we must find approaches that balance responsibility between a community that is unpaid and overwhelmed and a group that uses so much open source code that reviewing it all is most likely untenable. However, what is certain is to make OSS and proprietary code more secure, we will need strong partnerships between the open source community, private software enterprises, and federal agencies.