We need to talk about Product Security
Introduction
In the last few months we have seen a staggering number of trivially exploitable vulnerabilities in Internet-facing products. In addition to numerous vulnerabilities in secure file transfer products, we have seen recent critical vulnerabilities in Internet Gateways (CVE-2023-3519), MDM Solutions (CVE-2023-35078), VPN Servers (CVE-2023-27997) and Application Servers (CVE-2023-38203). In all these cases active exploits appeared before, or very soon after, the issues were disclosed by the vendors.
These products are designed to be Internet facing and yet they seem to be dropping like flies. So what is going wrong? Why are so many vendors continuing to sell security critical products that contain trivially exploitable vulnerabilities?
Adverse Selection
Secure product development is not rocket science. In reality it is not that difficult to select a set of technologies, people, processes and tools that will reliably and consistently turn out secure and trustworthy software. Delivering a level of security assurance at which an exploitable vulnerability will be a rare occurrence is definitely not out of reach. The key problem though, is that delivering this level of security assurance is very expensive.
Security assurance is also very difficult to measure. In practice, it is almost impossible for a typical corporate buyer to get an accurate view of the real level of security provided by a given product. The set of tools that are usually employed in this situation include vendor questionnaires, product certifications, product reviews, process compliance, vendor security statements etc. In the vast majority of cases these tools are ineffective, they will generally not provide any real insight into the level of security provided by a given product.
If you need convincing about this then just apply your standard purchasing process to the products with the vulnerabilities listed above. Every one of these vendors will have polished responses to every security question you can ask and will hold all of the security certifications needed to convince you their products are secure, yet every one also has has trivially exploitable vulnerabilities in their products. A little extra digging will also show that these vulnerabilities are far from isolated incidents.
So how can so many products, that look like they should be secure, turn out to have this type of vulnerability? The problem really comes down the way we try to measure security, there are two main reasons why this tends to go badly wrong:
Time: If you look at the above vulnerabilities then it is clear that many of the vulnerable components were actually developed many years ago, and often by smaller companies that were acquired by the current vendors. Most vendor security questions and certifications relate to current development practices, in many cases this tells you almost nothing about the process that was used when core parts of the product were designed and implemented.
Depth: Weaving together an effective secure development process is part art and part science. Attempting to determine the effectiveness of such a process using a small number of canned questions is never going to give a very precise answer. While it is possible to derive a set of process questions and metrics that apply to a specific product and technology, the typical vendor security questionnaire tends to focus on high level process questions. This just doesn't provide enough technical resolution to be able to differentiate between those vendors that focus on security and those that don't.
The combination of security assurance being expensive and almost unmeasurable creates a situation that economists call adverse selection. Vendors that prioritise security over features and time to market will actually tend to look less competitive than those that push out lots of features quickly. This leads to an effective 'race to the bottom' where the market will only deliver specific security features that customers need, less measurable aspects such as overall assurance and resistance to common vulnerabilities will tend to be optimised away.
The net effect of this is that we end up with a range of product offerings that may all look superficially secure but, in practice, the vast majority of the components that make up these products were probably not designed or implemented with security in mind. This means that critical vulnerabilities, and exploits, are almost an inevitability.
Follow the Money
A data breach can be a disastrous event for an organisation. The impact also extends to any downstream organisations that suffer from a supply chain compromise and to any end users or customers that may have personal data compromised. The one party that doesn't really feel any pain here is the vendor of the component that was exploited. Other than having to quickly issue an unplanned patch, the vendor will tend to walk away relatively unscathed even if their product was responsible for a compromise that impacted millions of people.
So are the vendors to blame for the ongoing data breach epidemic?
Any commercially succesful software vendor will do exactly what it takes for them to build a sustainable, profitable, business. Nothing more, nothing less. Those that try to deviate from this will tend to disappear very quickly. If the vendors are not seeing a very clear business case for something then they simply will not do it. Unfortunately, the adverse selection problem mentioned above means that, for most vendors, more features and faster releases are simply more profitable than slowing down and focusing on improving their security, so that is exactly what they do.
There are some vendors that genuinely do need to maintain a solid level of security to maintain their market positions but these are very much the exception rather than the rule. For the remaining 95% of the industry, the business case for robust security is generally very weak. Most vendors know that if they invest in security assurance rather than new features and more releases they will end up in a worse position relative to their industry peers.
This situation is a problem for vendors that actually do want to deliver more secure products. It makes it difficult to build an internal business case for a sustained investment in security, this can lead to security initiatives being driven as grass roots projects without the level of investment or executive support that they really need to be successful.
There is no doubt that many vulnerabilities are caused by vendor mistakes, but if the market demands features then the vendors will deliver features. The reason that vendors keep delivering products with relatively low levels of security assurance is, quite simply, because everybody keeps buying them.
Cognitive Dissonance
Now consider the following:
Question: Who is responsible for ensuring that there are no design or code level vulnerabilities in the software components you deploy?
Answer: That's the vendor's problem, they create the software and we expect them to make sure it is secure.
On the surface this seems like a perfectly reasonable position but, in a world where vendors do not have a strong incentive to deliver secure products, it is a recipe for disaster. You are trusting the security of your organisation and your customers to a vendor with no incentive to protect you. In other words: you are paying a fox to guard your hen house.
One day the inevitable will happen and some of your hens will disappear. You will then need to install a patch and let the updated fox (v1.0.1) continue to guard your remaining hens. The fox doesn't mind, he still gets paid.
The bottom line is that current best practices for secure software selection are simply ineffective. They are far too focused on selecting software that superficially looks secure rather than software that actually is secure. Unless this changes we are going to continue to live in a world of never ending network and data breaches.
How can we fix it?
This situation can be changed. If organisations put more focus into effectively assessing the level of security provided by the components they deploy then it would improve the security of those organisation's whilst also starting to apply more pressure on vendors to up-level their own security engineering practices.
The first critical step is for organisations to change the way they look at selecting software components. It is time to stop assuming that vendors will deliver secure components, doing so amounts to building your organisation's security strategy on a foundation of wishful thinking. Every product, from every vendor, should be assumed to be riddled with critical vulnerabilities until something demonstrates otherwise. Unfortunately this will tend put your starting position much closer to reality.
The relative priority given to secure product selection also needs to be considered. The fundamental level of security assurance provided by deployed software products is the basic foundation of any secure environment. If you don't get this right then no amount of additional defensive tools, services, or processes are going to keep your network safe. Secure product selection really does need to be one of your top priorities.
Looking Ahead
We are currently working on a number of projects that are related to helping organisations to identify and mitigate risks from insecure software. This is the first in a series of posts in this area so stay tuned for more, including:
Future blog posts covering specific processes and tools that can be used to assess the real-world level of security provided by a given product or technology.
We are looking into developing a supporting set of product, platform, and technology specific vendor questions that are designed to provide real technical insight and actionable data on the level of security provided by commercial or open source products.
We have an active engineering project that is looking at feeding product security insights into our EdgeScope tool to provide a continuous forward looking indicator of likely areas of risk from across your whole attack surface.
If have any thoughts or suggestions on areas of focus or future blog posts related to any aspect of product security then please let us know.
We also offer a Product Security Assessment service that is designed to provide meaningful real-world data on the level of security assurance provided by a specific product. So if you are looking for more insight into the security of your deployed infrastructure then get in touch.
