NetScaler CVE-2023-3519: Exploit Campaign Analysis

Vulnerability Patching
Written By Ian Wright

Introduction

Several months ago the MOVEit attacks provided a frightening demonstration of how an Internet-wide automated exploit can enable attackers to obtain an immense volume of data. In this post we are going to take a closer look at a more recent global exploit campaign, this time the target was Citrix NetScaler appliances.

Rather than do a deep dive into the specific exploit being used we are going to try to reassemble the series of events that took place during the attack to see if this sheds any light on exactly how these global exploit campaigns are being run. The following timeline provides some high-level context on the underlying vulnerability disclosure and existing coverage of the exploit campaign:

Following the Bread Crumbs

The specific exploit we are looking at uses CVE-2023-3519 to gain access to the NetScaler, it then installs a web shell and adds a malicious credential harvesting JavaScript component to the NetScaler's login page. The changes to the login page are interesting, this update changes the file timestamp on the page which gives us a way to easily determine the exact point in time at which the exploit took place. The addition of the JavaScript reference also provides a reliable indicator that a specific NetScaler has been compromised.

By cross referencing the indicators of compromise against the exploit timestamp and the NetScaler version being used, we were able to recreate a precise granular timeline of the global attack, here is how it unfolded:

  • The first wave of the attack started at 11:40 UTC on Sunday 6th August. This wave lasted for 8 hours and specifically targeted NetScaler servers running version 13.0-58.32. The attack was not selective, the majority of Internet-visible NetScalers running that specific build were compromised during this wave. Given the observed ordering and timeline of this wave it seems likely that the attackers were simply running a randomised scan of the full IPv4 address range to detect and exploit target servers.

  • The next step started at 16:00 UTC on Friday 11th August. During the next few hours the attackers appeared to be testing exploits against different NetScaler versions. They conducted a single test attack against one server running each of the builds: 12.1-63.24, 13.0-61.48, 13.0-71.44, 13.0-79.64 and 13.0-86.17. Note that the only outcome we can observe is a succesful NetScaler compromise, so it is possible that the attackers were trying attacks against other builds but none of these were successful.

  • Two days later, the next wave started. At around 16:00 UTC on Sunday 13th August the attackers started a 12 hour wave of attacks, this time the attacks were targeting NetScaler versions 13.0-47.24, 13.0-58.32, 13.0-61.48 and 13.0-83.29. This wave of the attack used a different domain to host the credential harvesting script. Many of the 13.0-58.32 NetScalers that had already been compromised during the first wave where actually compromised again by this wave, this resulted in many servers having two different script tags added to the login page. It isn't clear if this double-compromise was a mistake or if it was done to guard against the possibility of the infrastructure hosting the scripts being taken off line.

  • The third wave of the attack started the next day, just before 18:00 UTC on Monday 14th August. The wave ran for 10 hours and targeted five additional NetScaler builds: 12.1-62.27, 13.0-64.35, 13.0-87.9, 13.0-76.31 and 13.0-79.64. 

  • A similar pattern continued over the course of the next few weeks. An ongoing mix of low-volume testing against specific builds and then additional attack waves targeting those builds. The combined attacks targeted all of the following NetScaler builds:

    • 12.1-55.18, 12.1-56.22, 12.1-57.18, 12.1-58.15, 12.1-59.16, 12.1-60.19

    • 12.1-62.27, 12.1-63.24, 12.1-65.15, 12.1-65.25, 13.0-47.24, 13.0-52.24

    • 13.0-58.32, 13.0-61.48, 13.0-64.35, 13.0-71.44, 13.0-76.31, 13.0-79.64

    • 13.0-83.29, 13.0-86.17, 13.0-85.19, 13.0-87.9

  • The malicious credential harvesting scripts were hosted on the following domains:

    • jscdn.biz, jscloud.biz, jscloud.ink, cloudjs.live

    • cloud-js.cloud, jscript.club, cloud-js.cloud

  • The attacks slowed and then stopped during October. The last observed compromise took place on Wednesday the 18th October, the final victim was a US based air freight company. 

Observations

In order to reliably exploit this vulnerability the attacker needs to know the exact layout of the target host stack, this tends to vary slightly between builds so any exploit code needs to be fine tuned to target a specific NetScaler build. The use of exploits for specific builds seen during the various waves of the attack leaves a clear imprint of the way the attackers were testing and refining these exploit variations.

When the attack took place there were over 60,000 Internet-visible NetScaler instances, of which over 10,000 had not yet been patched against CVE-2023-3519. In total this attack compromised around 650 NetScaler instances. The need to develop and deploy a specific exploit for each build appears to have been the biggest factor in reducing the number of impacted servers.

While the majority of instances using the targeted builds did end up being compromised, there were several thousand additional unpatched servers that appear to have escaped simply because they were using builds that were not selected by the attackers. This turned the attack into something of a build based breach lottery.

This is a clear demonstration of how a small technical detail can have a significant impact on the scale and rate of any real-world exploit, it also shows why certain higher level vulnerabilities that provide a direct vector for code injection without having to worry about internal host-specific memory layout details (such as deserialisation vulnerabilities) are very likely to lead to broader and faster exploits. If the attackers had been able to develop a reliable cross-build exploit then it seems likely that this attack could have compromised a good proportion of the 10,000 unpatched NetScalers.

The time of day at which the attack waves took place is interesting. While this is sometimes used to guess the attacker’s timezone and location, we suspect the specific timing of these attacks was carefully planned. By timing most of the waves so that they ended at around 04:00 UTC the attackers ensured that the newly installed credential harvesting exploit would be likely to catch a high volume of morning employee logins from organisations based in Europe and the US. Furthermore, in many cases the exploits would already have captured those credentials by the time the security or operations teams started work, even if the breach was then quickly detected and fixed it means the attackers would still have obtained a significant number of user credentials.

The most ominous aspect of this attack is the way it ended. On 10th October Citrix disclosed another critical NetScaler vulnerability, CVE-2023-4966. This issue does not provide a direct route to code injection but it does allow an attacker to collect valid session identifiers that could then be used to hijack existing authenticated sessions. An information leak may not seem as bad as a code injection vulnerability, but in this case it may be worse.

A NetScaler has little inherent value to an attacker, the real goal is the user data and internal resource access that a NetScaler can provide. As CVE-2023-4966 provides a much more direct route to gaining access to a user session, and the exploit doesn't require any build-specific tuning, it is easy to why why CVE-2023-4966 presents a much more tempting exploit target.

The observed exploits of CVE-2023-3519 stopped soon after CVE-2023-4966 was disclosed. This might be a coincidence, but it probably means that the attackers did not actually stop, they simply shifted focus. 

Estimated Impact

The exploits we have seen using CVE-2023-3519 are likely to have been the first step in a series of events that will culminate in future ransomware demands. As with the earlier MOVEIt attacks, the true scale and implications of this type of attack can take many months to become apparent. While the number of compromised servers looks relatively small when compared to the total number of deployed NetScalers, this still represents several hundred organisations around the globe that, right now, could well be in a position where the attackers are quietly moving through their networks and collecting sensitive data for a future ransomware demand. 

If just 10% of these organisations end up being ransomware victims, and we use a mean recovery cost of $1.82M, as suggested by Sophos, then the overall cost of this single attack could amount to over $100 Million. In reality there are simply too many variables and unknowns here to provide an accurate projection, but this type of estimate can still be instructive as to the potential magnitude of the cost.

Missing Patches

Around mid-October we reached out to a number of organisations that owned compromised NetScalers. Out of curiosity, we also used EdgeScope to take a look at the other infrastructure of those organisations to try and get a sense of their approach to applying security patches. The results were very mixed:

  • We saw a number of organisations that had patched some, but not all, of their NetScalers. The unpatched instances had then been breached. This tends to be larger organisations that do have established security processes, but in the rush to quickly deploy patches across a complex environment had simply missed one or more vulnerable instances.

  • Many organisations were consistently applying patches, but were simply doing so too slowly to avoid being breached by an attack that occurred within a few weeks of a vulnerability disclosure. As CVE-2023-3519 was well known to have been actively exploited, this does suggest that these organisations are not correctly prioritising patches for vulnerabilities that are at risk of imminent exploit.

  • There were several cases where the compromised hosts appeared to have been abandoned. Here we saw older NetScaler builds, along with other indications such as historic DNS entries and long-expired TLS certs. In same cases these hosts were still sitting in corporate data centres so the compromises could still have been used to gain access to the organisation's network.

  • In certain sectors we saw organisations that clearly had work to do on their patching processes. These tended to be small legal firms, niche IT service providers, municipally owned corporations and (worryingly) many healthcare organisations. In these instances we saw a general lack of patching that resulted in some of them being vulnerable to this issue along with several other issues currently listed in the CISA known exploited vulnerabilities catalog.

Consistent, effective and risk-driven patch management is something that is clearly a significant challenge for many organisations. While there are other factors (such as the early disclosure of exploit details and missing NetScaler compiler mitigations) that helped to enable this attack, issues around patch management do appear to have been a key factor in influencing which organisations where breached.

Looking Ahead

As we head into 2024 it seems inevitable that that flow of critical vulnerabilities in Internet-facing infrastructure will continue. The current financial incentives for ransomware groups mean that it is equally inevitable that we will see a sustained focus on quickly exploiting newly disclosed vulnerabilities in order to gain access to a large number potential targets.

Based on our observations of this specific attack, there are some very clear steps that organisations can take to defend themselves:

  • Focus on consistent risk-driven patch deployment: The one key lesson here is to ensure that threat intelligence related to active exploits is used as the basis for prioritising security patch deployment.

  • Understand your attack surface: The indiscriminate nature of these attacks means that leaving a single host unpatched can lead to a breach, organisations need to have a clear view of all Internet-visible product instances to ensure that vulnerabilities are fully patched.

  • Remove obsolete/unused infrastructure: Abandoned infrastructure is an easy target for this type of attack, anything that isn't being actively used and maintained should be quickly removed.

  • Take steps to ensure that any service providers in your supply chain are applying the above practices: Based on our findings we would strongly recommend technical validation of this rather than relying on vendor statements, the unfortunate reality is that many service providers, including some security vendors, do not appear to be keeping on top of critical patches.

The speed and consistency of patch management that we typically see when looking at this type of vulnerability is concerning, many organisations are simply not moving fast enough to avoid falling victim to rapidly deployed automated exploits. If ransomware groups continue to focus on this type of attack then the level of ransomware activity we have seen during 2023 could well turn out to be the tip of a hugely expensive and disruptive iceberg.

How Can We Help?

Our ongoing research into attack surface analysis and active exploit campaigns is used to ensure that our security services provide focused and effective protection against current real-world threats. Organisations looking to defend themselves against the type of exploit campaign discussed in this post may be interested in:

  • Surface Guard: Combines current threat intelligence and sophisticated attack surface management technology with skilled vulnerability triage to continually monitor an organisation's attack surface for unpatched actively exploited vulnerabilities.

  • Surface Scan: Uses our powerful attack surface management tools along with skilled analysis to provide a point-in-time assessment of an organisation's attack surface and any high risk vulnerabilities. SurfaceScan engagements can also be used to evaluate the security posture and patch-management practices of organisations in your supply chain.

If you would like any more details on these services, or have any comments or feedback on any of our blog posts, then please get in touch.

Ian Wright
Previous

Dear Open Source, Can we ever trust you again?
Next

Threat Driven Security