Dear Open Source, Can we ever trust you again?
“Yet across the gulf of space, minds that are to our minds as ours are to those of the beasts that perish, intellects vast and cool and unsympathetic, regarded this earth with envious eyes, and slowly and surely drew their plans against us.”
Introduction
The recently discosed CVE-2024-3094 is not just another vulnerability. This is not a mistake that resulted in a security issue, it is deliberate malicious sabotage. Let's take a closer look at the XZ backdoor.
What happened?
Over the course of two years a carefully orchestrated social engineering attack was used to gain write access to an open source compression library used by OpenSSH. An elaborate set of obfuscated updates were then made that resulted in that library adding a backdoor to OpenSSH. The backdoor enabled an attacker using a specific authentication key to execute arbitrary commands on the server.
At the time of writing there are 26 million Internet-visible SSH endpoints and over half of these have banners identifying themselves as OpenSSH servers. Fortunately the backdoor was detected before it found its way into many production systems but this is still a significant wake up call as to the potential impact of this type of attack. This could have been much worse.
While it is clear that the attacker was patient and technically competent, there isn't very much else we can say about who the attacker really was. Possibilities range from a state sponsored group through to a single creative individual. While the speculation will be endless it is unlikely we will ever know who carried out the attack. We also don't know why the attack was carried out, the intent may have been to target a specific person or organisation, it may have been to get the backdoor deployed to millions of hosts, it may have been done just to make a point.
Implications
The novel quoted at the top of this post, H. G. Wells "The War of the Worlds", depicts a scenario where an alien race spends many years planning an attack on earth. The attacking robotic intruders were fired at the earth (or pre-buried, if you prefer the movie version). As you might expect, our fictional alien invaders sent quite a few of these intruders. After all, if you are going to spend years planning a campaign and developing the capability to deliver a malicious payload, then it would be strategically inept to bet everything on a single attempt.… Wouldn't it?
The obvious question now is if similar tactics have already been used to backdoor other components. And, if so, how can we find them?
The majority of the time and effort that goes into an attack like this is spent on preparing the ground: getting people into trusted positions and ensuring that you have the right hooks and features in place to be able to commit an obfuscated backdoor that won't be immediately discovered. At that point, the attacker can pick their moment to actually submit the payload and hope it finds its way into a production environment before it is found.
This alone makes looking for similar issues something of a challenge. It is not as simple as looking for obfuscated code or scripts as, in most cases, these will be very well hidden or simply won't exist until a short time before an actual exploit.
It is important to remember that, unlike most other types of code level security analysis, this is a scenario where a vulnerability is carefully designed to be hidden. At the moment the open and closed source worlds are struggling to find vulnerabilities that were added by accident. Finding issues that are deliberately concealed is an order of magnitude more difficult. In reality, it is essentially impossible to do this with any degree of reliability.
The social engineering elements of the attack are genuinely frightening. This involved a hobbyist working on his project who was subject to a coordinated campaign that applied pressure from one side whilst simultaneously offering help and support from another direction. In this scenario it is difficult to see how any reasonable person wouldn't take up the offer of help. None of us are invulnerable to this type of coordinated manipulation.
So here we have an adversary that is technically competent, able to spend years on a single attack, is able to pick the timing and nature of their attacks to avoid detection and is well versed in using social engineering techniques to achieve their goals. Their targets will often be part time hobby developers and package maintainers. This is not a fair fight.
There will be others
This type of attack can have a high return on investment. While it clearly took time and technical skills to conduct, a secret backdoor in a component like OpenSSH could be extremely valuable. You can look at this attack from the perspective of a ransomware group looking to extort money, a nation state looking to obtain access to sensitive people or organisations, or even a lone activist, in all cases the potential results easily justify the time and expense that was invested in the attack.
There is simply not enough data to try and speculate on the potential scope or scale of these attacks. But, on balance of probability, it seems highly unlikely that this is just a single isolated incident. At this point in time we simply do not know if there are other open source components out there that already contain this type of backdoor, or other situations where attackers have got themselves in a trusted position and are just waiting for the right time to conduct an attack.
Can we fix this?
At this point it is not clear what a fix for this type of attack even looks like. Adding technical measures to detect suspicious code is one obvious approach, but the attacker will nearly always have prior knowledge of these measures and is likely to be able to conduct their own testing to determine what will and will not be flagged as malicious.
Social engineering risks can be reduced through regular training and guidance. Making this work in an open source world could be challenging but this may still turn out to be one the best ways to prevent future attacks. This specific event will already have served to raise awareness of the risks, this will only be a short-term effect though.
Initiatives like the CISA's work on SBOM (Software Bill of Materials) can help to provide more clarity over the composition of software components, while that does provide more traceability it doesn't really help to prevent the attacks in the first place. We can also expect the attackers to be carefully watching this space to see if this provides any additional insight into exploitable component dependencies.
SBOM could also provide valuable input for those building high assurance environments. If you have multiple layers of defence then ensuring that there is minimal code shared between the layers can reduce the risk of a single vulnerability or backdoor being used to compromise high value data.
Defensive Steps
It seems clear that this threat is not going to go away. While the open source community is already looking at what they can do to prevent and detect these attacks, the most pragmatic immediate step that most of us can take is to accept that these supply chain risks exist and to take steps to reduce our own level of exposure.
In practice, this means that the best step most of us can take is to make sure that Internet-visible components are kept to an absolute minimum. As with many types of threat, general attack surface simplification and reduction doesn't eliminate the risk but it can provide a significant real-world improvement.
Timing is also an important element of this type of attack. Getting a backdoor into a stable component release will always be more challenging because it needs to go through more steps, and will typically be seen by more people and subject to more thorough testing. That means sticking to supported versions but avoiding the bleeding-edge could help to reduce your exposure.
Conclusion
Overall, it looks like this attack was a near miss. However, it seems highly likely that we will see similar attacks in the future and these could have a much greater impact. There is also a clear and somewhat unsettling risk that other components may already contain similar backdoors that are being actively used in targeted attacks.
Raising the priority of any projects relating to defence in depth or attack surface reduction is probably the best short term mitigation option for most organisations. If you need any help working out how to reduce your exposure to Internet based attacks, then get in touch.
