The Importance of Risk Management
Introduction
In October last year Ivanti disclosed two CVEs that impact their Connect Secure VPN product. Both of these are described as being high severity (CVSS 7.5) and both impact Ivanti's Connect Secure product (previously known as PulseSecure) as well as some other Ivanti product lines.
As we are now five months down the line, let’s take a look at how the patch deployment is going.
CVE risk assessment
Before we look at any patching data, let’s take a look at the CVEs. Getting a baseline understanding of the risk profile of the vulnerabilities should help to set some initial expectations on how quickly they should be patched. The CVEs we are looking at are CVE-2022-35254 and CVE-2022-35258
At first glance, there isn't a lot of technical detail to go on. Ivanti's security advisory has an impact section that states:
"Denial of service: normal operation of the Ivanti Connect Secure (ICS) application will resume once the attacker stops sending malicious traffic."
Both CVEs also have identical CVSS scores, a vector of AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H giving a base score of 7.5 (High). This essentially tells us that an unauthenticated network based attacker can have a high impact on the availability of the service. Taken in combination with Ivanti's comment that the "application will resume once the attacker stops sending malicious traffic" this is starting to look like a relatively minor Denial of Service issue. It being somewhat less serious as the attacker would need to sustain a DoS attack to maintain any level of service impact.
A little more digging does provide some additional context from the vulnerability reporter:
Controlling EIP generally puts an attacker more than half way towards achieving remote code execution. Even if it is difficult to get enough control over the memory to actually execute arbitrary code, this now means that our minor looking DoS is starting to look like a more significant risk.
A little more digging leads to the following comment on the same issue:
“Recently Marco Ortisi has found for RedTimmy Security a DoS bug in the latest version of the VPN solution Pulse Connect Secure (PCS). The exploit crashes the remote service. After the crash a watchdog process is in charge to reload it again within few seconds”
Source: http://web.archive.org/web/20221006144113/https://www.redtimmy.com/tag/denial-of-service/ [Web archive link as the origin site was not available at the time of writing]
This comment aligns with Ivanti's comment about the service recovering once the DoS attack stops. If you look at this in terms of achieving code execution though, the same mechanism that reduces the impact of a DoS vulnerability may now significantly increase the risk of the vulnerability being exploited.
In a situation where the attacker has limited control over the memory layout it may be only possible to achieve code execution in a small number of instances due to random elements that the attacker cannot account for. In other cases, the target process may just end up crashing. If there is a mechanism that will continually restart the vulnerable process then, even if the exploit only works 1% of the time, an attacker can significantly increase the chances of compromising the target by continually repeating the attack.
Having collated the public information on these CVEs we now have two, somewhat different, views on the likely risk level. It is of course possible that Ivanti has throughly investigated the exploit routes and concluded that there really is no known way to achieve code execution. Even so, the very fact that this issue may involve memory corruption does somewhat change its risk profile.
At this point there is always the option of seeking technical clarification from the vendor, or digging into the issue further to see if code execution can really be achieved. However, as the patch is already available, the pragmatic approach would be to make a balanced assessment of the risk and then update the vulnerable product in an appropriate timescale.
I would characterise this as something like: "a DoS issue that, in time, may lead to a risk of code execution". Even taking the CVSSS score of 7.5 at face value this is clearly something that should be patched reasonably quickly.
Measuring patch deployment
As is evident from Ivanti's security advisory, there are number of supported branches of the Connect Secure product line and there is a fixed version available for each supported branch.
This means we can't just use a single version range to differentiate patched from unpatched deployments. Something we can use is the build age: any builds deployed before the advisory was published will not contain the patch. Our EdgeScope product uses a set of pattern matching heuristics to differentiate between product builds and to work out the age of each build, we use this to match CVEs to specific product builds in scenarios where an explicit version number is not visible.
We can use this approach to assess the build date of any deployment of Connect Secure. Using this data gives us the following view of current deployments (as of mid-March 2023):
As there are a relatively high number of builds covering various point releases of all the supported branches, we have put the builds into buckets and then counted the number of distinct IPs that host a build that is before (blue) or after (amber) that date.
The counts include deployments of the original PulseSecure product, the later Ivanti branded releases running as part of the Ivanti "legacy stack" and the newer 2x.x releases running as part of Ivanti's "modern stack".
The advisory for the CVEs we are interested in was released on 13th October 2022. The chart shows us that there are 26,736 deployments with build dates before 1st October and 7,338 deployments with a build date on or after 1st October.
This data tells us that less than 25% of the discoverable Internet facing deployments have been patched against two high severity CVEs that were released over five months ago. This is a worryingly low number. If we add in even a theoretical possibility that one of these could lead to code execution in a VPN appliance, then that number is nothing short of terrifying.
What went wrong?
A quick analysis of the unpatched IP addresses (using the relationship graph behind EdgeScope) lets us associate the most likely owner with each unpatched IP, the result is not pretty. That list contains numerous major financial and healthcare organisations, a significant number of US and UK government departments, including some with cyber security responsibilities, not to mention a sizeable set of airlines, hospitals and multinational manufacturing companies.
In other words: there are a lot of exposed organisations that really should have applied these patches very quickly.
One explanation is that these organisations have incorrectly prioritised these CVEs. It seems quite plausible that somebody saw the CVSS rating stating that this only impacted availability and then just added a task to their backlog to install the patches at some point in the future.
So is the CVSS rating wrong? It appears as though Ivanti assessed the issue according to the known impact of the vulnerability rather than using the (theoretical) risk of code injection. In some situations this is the only reasonable course of action as proving that a DoS really is only a DoS amounts to proving a negative. This is also exactly how CVSS scores are supposed to be assessed. "analysts should constrain impacts to a reasonable final impact which they are confident an attacker is able to achieve" https://www.first.org/cvss/user-guide.
I suspect the real reason why so many organisations are being slow to deploy this patch is due to a general lack of technical depth in their vulnerability risk assessments. The CVSS base score provides one very specific measurement on the technical severity of a vulnerability. CVSS simply does not have the means to convey information like: "This is a DoS issue, but in time somebody may be able to work out how to use this to achieve code execution".
The key takeaway here is that the real scope and impact of a vulnerability can be more nuanced than a single CVSS base score. It is always advisable to monitor public information sources to ensure you have fully understood the overall shape and potential impact of a vulnerability.
And perhaps most importantly: If it’s on the Internet, and there is any doubt about the scope of the vulnerability, then just get it patched. If you aren't sure if it is reachable from the Internet then get in touch, we can probably help you with that.
Tip of the Iceberg?
There is currently a trend towards vendors providing less technical vulnerability information in security advisories or vulnerability disclosures. The rationale behind this is to reduce the risk of the underlying vulnerability being analysed and exploited before the patch can be widely deployed. A potential downside of this is that it means end user organisations have less context on which to base a decision as to how quickly to deploy any patches. In many cases a single CVSS base score is all they get to go on.
An additional challenge is that security processes tend to follow a linear workflow process. So a vulnerability will be assessed once based on the initially available information and then a patch timeline will be defined. In a world where the risk level can change dramatically in the days and weeks after a vulnerability disclosure, this type of fixed workflow can result in an organisation taking far to long to fix high risk issues.
The above tweet, that adds some significant detail on the Ivanti CVEs, was sent over a week after the vendor advisory. It would be interesting to know how many of the currently unpatched organisations had already prioritised the patches at that point, and never reviewed their plan to account for the new vulnerability details.
Time for a check up
Effectively prioritising patch deployment is one of the most critical aspects of security risk management for any organisation, and there are some clear indications that not everybody is getting it right.
It is easy to fall into the trap of believing patch management is a ‘solved problem’ as you have ‘a process for that’. If you haven’t looked into how effectively your organisation is managing evolving risks than now might be a good time to check on how well that process is really working.
If you need an expert opinion on how your organisation is doing then get in touch. We offer a range of consulting and monitoring services that can help to measure and improve your patch management processes.
