A Million Weak SSH Keys?
“A secret is most valuable when it remains a secret.”
Introduction
Sometimes a specific CVE just looks a little bit more interesting than the others, a few days ago this one made its way into our vulnerability feed:
CVE-2023-37426: EdgeConnect SD-WAN Orchestrator instances prior to the versions resolved in this advisory were found to have shared static SSH host keys for all installations.
The noteworthy point about the above CVE is that this whole class of vulnerability (product instances using the same SSH host key) should be easily detectable. In theory it should be possible to just take a database of Internet visible SSH host keys and then automatically detect any cases where we see several instances of a given product sharing the same host key.
Automated vulnerability detection is always interesting, so let’s see if this actually works.
Welcome to the Real World
When it comes to Internet data analytics, real world measurements are often wildly different from intuitive expectations. A quick analysis of currently known Internet-visible host keys immediately shows that automated detection of shared keys is going to be a little bit more problematic than expected.
The following table is an initial frequency analysis showing the number of IP addresses associated with every host key we have in our database:
There are clearly a lot of shared host keys out there. So what is going on? A look through a sample of these reveals a number of different scenarios that result in us seeing multiple IPs associated with a single host key:
Port/Service Redirection: A common pattern used by some network providers, especially residential ISPs, is to redirect incoming traffic for specific services to their own infrastructure. This means that a wide range of IP addresses can end up being redirected to a single server (or cluster of servers). It is these types of deployment that result in the table entries where we see tens of thousands, or even millions, of IP addresses that appear to have the same host key.
Multi-homed Hosts: If a network has multiple Internet links that accept incoming connections then we could again see cases where a single host key looks like it is being shared. These tend to result in a key having a much lower number of associated IPs though.
Clustered Servers: If you are managing a set of servers that are inside the same physical and logical security boundary then it can make sense to use the same keys on all the servers. This is something of a trade-off between key management overhead and security granularity but there are clearly cases where this is done. A typical example would be a high-availability pair, or larger set of centrally managed servers, where it is beneficial to push the exact same configuration to each end point.
Dynamic IPs: Many residential connections, and even some smaller business connections, use an IP that changes over time. This means that if we monitor SSH keys over a period of time then we will naturally start to see more than one IP associated with the same host key.
Actual Vulnerabilities: We know that somewhere in the above table, issues like CVE-2023-37426 are hiding out. These are the cases where we have a shipping product that is using a shared, and therefore potentially weak, host key.
Having established that there are number of different cases that result in host keys being shared across IP addresses, the interesting question is if we can somehow differentiate between these cases and filter out everything except the apparent vulnerabilities.
We now have a data analysis problem.
Pattern Recognition
Our first approach was to see if there are any likely characteristics that are unique to vulnerabilities like CVE-2023-37426. If we can derive a suitable set of heuristics then we might be able to filter out all the other cases and identify the interesting keys.
An initial observation was that instances of a vulnerable product are likely to appear on a random-looking set of IPs that are fairly well distributed across the Internet, some of the other cases of shared keys might be much more tightly clustered around specific IP ranges. So can a simple measure of statistical uniformity be used to find vulnerabilities?
This approach showed some promise but didn't really give us the needed results. Some of the cases outlined in the last section (such as ISP service redirection) do exhibit very strong IP clustering, but there there still many cases where the distribution metric alone couldn't really differentiate between vulnerabilities and legitimate deployments.
The next approach was to try a process of elimination. If we can find a set of characteristics that are specific to the legitimate cases where host keys are shared then we can eliminate these and will just be left with the interesting ones. To do this we derived a number of metrics that attempt to find similarities between the set of IPs associated with each host key. These included:
Our existing measure of numeric uniformity of the IPs.
A similarity metric based on the containing IP block, announced BGP routes and associated ASNs.
A similarity metric based on WHOIS records and contacts associated with the IPs.
A similarity metric based on patterns in DNS names that resolve to the IPs at which we found the keys.
A similarity metric based on patterns in Reverse DNS (PTR Names, Domains, and associated DNS servers).
We scored the IP set similarity for each shared host key using all of the above metrics and then used a weighted selection to pick the "best" score. Sorting the set of keys and putting those with the "worst" score first gave us an ordering that puts the keys with the highest numbers of seemingly unrelated IPs at the top. These keys are the ones that are most likely to be caused by product-level vulnerabilities.
Findings
We found problems. Lots of problems. The above filtering method was not perfect but it did remove the vast majority of the noise from the picture and it left a significant number of cases where keys were being shared between instances of the same product. Notably, there were some specific types of product that seemed to be relatively common in this filtered list, including:
VOIP Infrastructure
IP cameras and DVRs
Consumer/SMB Routers
NAS Servers
Audio and Video Media Servers
There were also some cases where specific sets of devices, such as Cisco Routers and Fortinet Firewalls, appeared to be using the same host keys even though they were in use by unrelated organisations. Our current best guess is that these were managed devices that had been configured and provided by third-party service providers. This is definitely not good practice, it means that any of the provider’s customers could obtain the host key being used by all the customers. Later connections from the provider to other customers could then be vulnerable to a man-in-the-middle attack that could potentially lead to network breach.
We also saw some cases where a single host key was shared between a number of products from different vendors. This seemed to be caused by several vendors branding and reselling the same generic product without checking to ensure that the base product was generating new host keys. While we will be following up with some of the vendors it is unlikely that we will be able to identify the actual vendor of the generic product so we are not optimistic about being able to get many of these issues resolved.
Shared Keys Considered Harmful
Analysis of the findings showed that some of the devices using shared keys also had a history of other high risk vulnerabilities, such as weak default passwords or hidden backdoor accounts.
This wasn’t really a surprise. The general product security hygiene problem that would result in a failure to generate a unique host key is likely to manifest itself in many other ways that could lead to many other types of vulnerabilities. In essence this means that (outside of legitimate clustering scenarios) the use of a shared SSH host key is a reasonably good indicator of weak product security.
Having this type of easily discoverable 'indicator of weakness' may actually present more real-world risk to a network than the typical man-in-the-middle threat that is normally associated with the use of a shared host key. It is somewhat analogous to waving a big flag with "lots of vulnerabilities here" written on it.
Vulnerability Detection
This type of vulnerability is an unusual shape as it isn't easy to scan a single host or appliance and reliably determine that it is using a shared host key. The analysis needs to include context from current Internet-visible keys and that set is constantly changing. The only reliable detection method is to continually compare host keys used by your Internet visible SSH end points with host keys used elsewhere on the Internet. The difficulty of detection may explain why we see so many shared host keys, if common vulnerability scanners don't detect these issues then it means they are much less likely to be identified and resolved.
Our EdgeScope Attack Surface Management product and our Managed Services now include continuous automated detection of shared SSH host keys. We also provide supporting analysis and vendor vulnerability reporting to ensure that any customer issues relating to the use of shared host keys can be quickly resolved. If you are looking for some help mitigating this risk for your organisation then don't hesitate to get in touch.
Next Steps
At this point we have barely scratched the surface when it comes to analysing all the potential vulnerabilities we identified during this process. We are going to continue working through these and will also be continuing to refine the detection heuristics. At this point we are not planning to publish the filtered list of shared keys as doing so may just increase the risk of these issues being exploited.
We are planning to release some follow up posts as our analysis progresses so stay tuned for future updates. In the mean time, we would strongly recommend that everybody includes some checking for usage of shared SSH host keys in their infrastructure, especially for Internet-facing servers.
If you do find any shared keys (other than expected server clusters) then it is worth reporting these to the component vendor and taking steps to ensure that the vulnerable component is not Internet-visible.
