MOVEit Exploit: Predictable and Avoidable?
Introduction
As the scale of the recent MOVEit exploit starts to become clear it is natural to start asking questions about what went wrong. We are going to take a brief look at MOVEit from a product security perspective to see how these vulnerabilities came to exist and if there were any early warning signs.
Let's start with a quick trip back in time…
May 2021 - A Problem Emerges
The first signs of a potential problem appeared two years ago. Between May and August 2021 there were three post-authentication SQL injection vulnerabilities disclosed in MOVEit: CVE-2021-31827, CVE-2021-33894 and CVE-2021-37614. Later that August a more serious pre-authentication SQL injection vulnerability was disclosed: CVE-2021-38159.
Rather than concentrating on the specific exploit approaches and patterns associated with these issues, let’s take a look at what these vulnerabilities tell us about the overall security maturity of MOVEit at that point in time. Looking through the publicly available details on these vulnerabilities there are some relatively clear conclusions that can be drawn:
The MOVEit code was constructing SQL queries using simple string concatenation, this is the classic cause of SQL injection that was first reported back in 1998.
Fixes were made to a number of vulnerable code paths, so the scope of the issue was wider than a single specific code flaw.
The fixes were implemented by attempting to escape user controlled data that was used to construct SQL queries.
The overall impression from the available data is that, at this point in time, MOVEit had very little protection against SQL injection attacks. These exploits did not have to jump through any loops to utilise edge cases in existing mitigations, simply because there weren't any existing mitigations.
It is worth noting that SQL injection vulnerabilities and mitigations had been well known for over 20 years at this point, so it does look like MOVEit's secure coding standards were not well aligned with known best practices.
The specific choice of mitigation approach is also instructive. To understand this let’s look at the current mitigation strategies for SQL injection vulnerabilities, the OWASP SQL Injection Cheat Sheet has a nice summary of these:
Defense Option 1: Prepared Statements (with Parameterized Queries)
Defense Option 2: Stored Procedures
Defense Option 3: Allow-list Input Validation
Defense Option 4: Escaping All User-Supplied Input
The MOVEit fixes implemented option 4. The OWASP page describes this as a "frail" approach that "should only be used as a last resort". This approach is not generally considered to be very robust because the correct escaping rules can vary depending on the back end database, the string encoding being used and the specific type of query being constructed. There is also a significant risk that, as the code evolves and changes, an escaping step for a new or updated query could be missed. In essence this approach can be effective but it is inherently fragile.
At this point the vendor needs a fast and efficient approach to mitigate some specific reported vulnerabilities so, even in light of the known limitations, choosing the escaping option and quickly resolving the known vulnerabilities was an understandable step.
However, this fix effectively leaves us with a ticking time bomb. At this point MOVEit does not appear to have a robust defense against SQL injection and, perhaps more importantly, there is now enough public information for anybody analysing the product to reach the same conclusion.
The only question now is if the vendor can implement and deploy a more robust protection method before additional vulnerabilities are found and exploited.
May 2023 - The Time Bomb Explodes
We now return to late May 2023, and it is starting to become clear that all is not well in the world of MOVEit deployments. Initial reports of detected exploits start to appear and it soon becomes clear that a new vulnerability has been found, exploited, and that a potentially massive amount of sensitive data has been compromised.
In this situation there is, once again, only one practical option open to the vendor: fix the specific vulnerability that is being exploited, release a patch and warn their customers as fast as possible. The vendor took this step and, to give them due credit, they did this very quickly and communicated the state of the issue openly and effectively.
Unfortunately this amounts to doing a good job of locking the stable door when the horses (or to be more precise: all their customer's horses) have already been stolen and were being led over the horizon by a ransomware gang.
Only a few days after the initial disclosure an additional fix was released for some additional pre-authentication SQL injection vulnerabilities, and we are now seeing reports of yet more vulnerabilities.
The number of vulnerabilities, and a brief analysis of the fixes, makes it clear what has gone wrong: MOVEit was still using the "frail" SQL injection mitigation that was added in 2021. In light of this, the recent exploit should not be a surprise. It was, in reality, nothing short of an inevitability.
How did this Happen?
There are two difficult questions that need to be asked here:
Why hadn't the vendor improved the SQL injection defences in the two years since the last vulnerabilities were reported?
Why are so many large organisations still deploying a product when there is a clear indication of an existing weakness?
While we might never have complete answers to these, there are some aspects of these questions that are worth commenting on.
Why hadn't the vendor improved the SQL injection defenses in the two years since the last vulnerabilities were reported?
Anybody that has spent any time in a commercial software development or product security role can provide some important context on this. One key point is that making sweeping changes to a widely deployed product is much more complicated than you might expect. Implementing a large scale change to the way SQL queries are processed, back porting some or all of these changes to a range of supported versions and then testing all supported combinations of product versions, configurations, and environments is definitely a not trivial engineering project. While I do not have any inside knowledge on this, it is at least possible that the vendor was working on these improvements but simply ran out of time.
The counter point is obviously that, given the potential customer impact, maybe the vendor should have made sure this happened much sooner.
Why are so many large organisations still deploying a product when there is a clear indication of an existing weakness?
This one is harder to answer, but there are some contributing factors:
Understanding the real-world risk implications of a product security gap is a skill set that is generally only found in the software development world. Most commercial organisations that are looking to procure a software solution simply don't have the skills or resources to be able to pick up on this type of risk indicator. Unfortunately this means many purchasing decisions still tend to be influenced more by marketing claims than factual assurance measurements.
The current Cybersecurity industry is heavily focused on detection and response rather than prevention; this introduces a risk that organisations put too much faith in their ability to detect and block exploits and then don't put as much focus on the more preventative aspects of security. This is a mistake, product security assurance is, and always will be, the fundamental building block of any secure system.
Software deployments in corporate networks tend to exhibit a high degree of “stickiness” that is to say, once deployed they tend to stay around forever. Even if an organisation does conduct due diligence when deploying new components it is much less likely that there will be a process for tracking this over time. Most organisations have “Vulnerability Management” processes, but “Assurance Management” is a term that is rarely used in the commercial world. This tends to mean that emerging indicators of risk, such as the emerging SQL risk in MOVEit, tend not to be evaluated or acted on by organisations that have already deployed the product.
Protective Steps
One obvious conclusion is that organisations need much better tools to measure and monitor the real world security assurance of the components they deploy or use as part of their supply chain. This is easy to say but, in practice, is very difficult to achieve.
Product certifications may look like the obvious answer here but history has clearly shown how difficult it is to successfully apply these in a commercial environment. The type of prescriptive approach used in the defense world, that dates all the way back to the Rainbow Series from the eighties, just doesn't seem to be practically applicable to the wider commercial software industry. Vendor RFPs or security questionnaires are another popular option. In reality these tend to contain standard questions to which all vendors have standard answers so these don’t often turn out to be very effective as a security assurance measurement tool.
There really is no “silver bullet” answer here, but there are some specific steps that an organisation can take to reduce the risks of this type of exploit.
Include an assurance evaluation step as part of your vulnerability management process. Don’t just apply patches, look into how vulnerabilities happened, how they were fixed and what this might mean in terms of future risks for your organisation. And never hesitate to ask vendors very pointed questions about why any vulnerabilities came to exist and what they are doing to reduce the risks of future issues.
Always consider the potential scope of an application level exploit. The reason why file transfer apps are such a popular target is that they have a short and fast exploit chain: A single exploit immediately gets a lot of valuable data. For any applications like this, where the application itself forms your only real layer of defense, make sure you single it out as a high risk deployment and take the time to investigate, and monitor any potential security risks or assurance gaps.
If you issue any form of RFP or vendor questionnaires then include very specific questions about past vulnerabilities and cross check their responses against current product security best practices.
Be realistic about the level of protection provided by your existing security tools and processes. Layered defences and effective detection and response are definitely valuable, but don’t lose focus on the need to maintain a good level of product security assurance and to keep your Internet facing attack surface to an absolute minimum.
Caveat Emptor
The bottom line here is that the ability to effectively gauge the level of assurance provided by a product should now be considered a key core competence for any organisation.
Doing this effectively increases the security of your network and also exerts a positive longer term influence on the market by ensuring that vendors that do deliver higher levels of security assurance will tend to be more successful.
