Patch or Be Damned
Introduction
Applying security patches is an endless task. It can be dull, thankless, complex, and often carries the risk of an unwelcome 2am phone call when a patch unexpectedly breaks a production service.
Applying security patches is also one of the most critical parts of any security process, if this isn't done quickly and consistently then it is only a matter of time before you get an even more unwelcome 2am phone call about a network intrusion or ransomware demand.
So how good is everybody at installing patches? Let’s look at some data and find out.
Measuring Exposure
Rather than trying to collect and combine a mountain of data covering a wide range of products, we are going to pick a specific sample data set covering a single product over a short time period. For this analysis we will be looking at Internet visible deployments of Microsoft Exchange Server during a six month time period: September-2022 through to February-2023.
To try to keep the analysis focused on real deployments we are going to limit the analysis to servers that meet all of the following criteria:
Use a supported version of Exchange Server (2019 CU11 or CU12, 2016 CU23 or 2013 CU23).
Have at least one security update installed during the test period.
Were present for at least three months during the test period.
This is to help to reduce the effect of non-production deployments on the metrics.
To quantify the patch time, we are going to calculate a days-at-risk metric for each server that meets the above criteria. This metric is simply a count of days that the server was missing one of the needed security updates. This is a relatively crude metric in terms of measuring real-world security risk but it should work for this analysis.
Show me the data
To start with, we measured the average days-at-risk across the whole data set, this tells us that the average Exchange server in our data set was unpatched for 58 days during the 180 day test period. Let's put that another way: the average Exchange server was in an insecure (vulnerable to a publicly disclosed vulnerability) state for almost a third of the time. This looks terrible, security is not supposed to be part time!
A single average is never going to tell us very much though, let's look at some other views of the data.
Ever wonder which country is best at applying security patches?
Average days-at-risk by country.
The level of variation in the above chart is interesting. The analysis is only looking at a single product during a single time period but it does still suggest that some countries are performing much better than others when it comes to keeping their infrastructure secure.
Proportion of servers by accumulated days-at-risk.
The above chart adds a lot more depth to the earlier average days-at-risk. It shows the proportion of servers by their accumulated days-at-risk.
Let’s start with the good news: The 30% of servers with less than 9 days accumulated days-at-risk does show that some servers are being well managed and are having security updates quickly applied. The bad news is the remaining 70%, there is clearly a long-tail when it comes to rolling out patches.
What’s the rush?
The data shows that patch deployment can be relatively slow, but is that really a problem? The short answer: yes, it is a problem. The hidden pitfall here is that the level of risk for a given vulnerability doesn’t stay the same over time. This might seem counter-intuitive: vulnerabilities have an assigned severity, typically a CVSS score, and this tends to be fixed. So what changes?
The majority of CVE disclosures are proactive disclosures by a product vendor, that means that the vulnerability isn’t known to the public at the point it is disclosed. When these issues are disclosed it takes time to develop and deploy an exploit. The specific time period is highly variable, sometimes vulnerabilities are exploited within hours of being disclosed, sometimes exploits can start to appear years later.
There are a multitude of technical and environmental factors that can influence the time and scope of any exploit attempts. In general though, as time goes by the risk of an exploit being developed and used will tend to increase.
Now look at the above chart again, that long-tail of servers with a higher days-at-risk are actually being exposed to a disproportionate level of risk. Quantifying this is difficult due to the number of variables involved in each vulnerability but if a patch isn’t installed quickly, certainly within a few days, then the real-world risk level can start to look ugly.
While there are a number of indicators that can be used to fine-tune a patching strategy, it isn’t always possible to reliably predict the timescale and scope of exploit attempts for a specific vulnerability. This means that the only generally effective strategy is to install any patches as quickly as possible and to closely monitor data sources such as the CISA’s Known Exploited Vulnerabilities Catalog to catch any cases where exploits appear for something that can’t be quickly patched.
Our EdgeScope product utilises the CISA data alongside CVE data from NIST’s National Vulnerability Database to give you a current view of the risk levels across your organisation’s attack surface.
Is there a better way?
Unfortunately, at least in the short term, there really is no substitute for quickly and consistently patching known vulnerabilities.
Understanding, monitoring, and reducing your attack surface, utilising internal analysis and detection tools, moving to a zero trust architecture, and making good use of threat intelligence sources are all measures that will put you in a better place. But all of these need to be applied on top of a consistent and robust patching strategy. If you don’t have one, you are effectively building on sand.
Longer term, we can only hope that vendors can take steps to significantly improve the level of security assurance in their products. Ultimately that is the only way that we will escape from an endless cycle of security patches and the associated windows of opportunity for cyber criminals.
At the moment though, it is clear that a lot of organisations are struggling to keep up with patch deployment. This exposes those organisations, and the people who’s data they hold, to a significant ongoing level of risk.
