If you had to explain it to someone outside cybersecurity, they wouldn’t believe you, but organisations including one of the world’s biggest banks and largest defence contractors have been compromised because they were too slow to apply a patch for a critical vulnerability. I was disappointed when it only took me 15 minutes to find a UK organisation who still hadn’t applied this patch today1, despite it being released over a month ago, and despite numerous warnings that ransomware groups are actively exploiting it.
Background
This all started on October 10th when Citrix issued a patch for a critical vulnerability (CVE-2023-4966, CVSS 9.4) in their NetScaler product. In a blog post, Kevin Beaumont described as the vulnerability as;
“incredibly easy to exploit”
“allows the bypass of all multifactor authentication controls, and provides a point and click desktop PC within the impacted victim’s internal network”
Kevin Beaumont doublepulsar.com
Apparently initial exploitation of this vulnerability was only used to target specific organisations, but after information about the vulnerability and the ease of exploit became more widespread, mass exploitation began, illustrated in the GreyNoise graph below.
What should happen
A widely exploited critical vulnerability is the scenario Cyber Security Professionals are paid to deal with (or prevent). The sequence of events after the vendor patch was released should be;
- Learn about the vulnerability from the vendor, a government alert, or in my case via social media.
- Confirm which of your systems are affected by checking your asset inventory
- Implement any mitigations you can (including restricting and/or monitoring access)
- Get an emergency change approved to patch the box
- Patch the box and apply any other vendor recommendations (in this case killing all active and persistent sessions)
- Check that the patch has applied and you’re no longer vulnerable
- Go back to what you were doing
What actually happens
- Learn about the vulnerability when either (a) your systems go down or (b) you receive an extortion demand from a ransomware group, whichever is quickest.
- Panic
The reasons behind this have already been written up by better commentators than me (h/t Kevin Beaumont) but I wanted to focus on one of the contributory factors he referenced, because it is SO crucial to effective cybersecurity, but it’s almost an afterthought for many organisations.
Asset management
Asset management isn’t sexy, it’s not something executives will come back from conferences excited about, but without asset management you can’t do security. Personally, I think there’s almost no part of cybersecurity you can do without effective asset management, but I’ll pick out four areas which would have been significantly contributory factors for organisations impacted by CitrixBleed.
Vulnerability management
Unless you know what you’re running, you can’t do vulnerability management. I track multiple sources of information to tell me what vulnerabilities have been published, which are likely to be or are being exploited. A lot of that information I can discard because I’m not running the affected system, but I only know that because I have (I hope!) an accurate asset inventory. The information which is relevant tells me which products are affected, the severity of the risk, and what the recommended action is.
A lot of vulnerabilities don’t get individual attention because there are just so many, and most can be mitigated by just staying up to date on all your software updates. Again, being confident in your coverage is vital, because the one system you didn’t know about (or forgot about) which won’t get patched is the one cyber-criminals will use to get into your organisation.
Logging and monitoring
Logging and monitoring are what gets you out of trouble if there’s a zero-day vulnerability in one of your systems, or a vulnerability which can’t be fixed by software updates (e.g., misconfiguration, malware on a device or a compromised user token). For logging to be effective, you need to be collecting it from all your systems, and how do you know where all your systems are? Your asset inventory.
Incident management
During incident triage and analysis, the scope and impact of an incident drives the response. If you don’t know what systems even exist (“I didn’t know we had a read only domain controller there!”) or what data is held on those systems (“Why does eu-filestore-1 have a payroll data dump on it?”) you can’t respond appropriately.
Asset Management basics
So having established that asset management is a fundamental, even foundational cybersecurity control there is a tendency for the response to be “can I buy an app that does that?”, to which the answer is “Yes, but you still need to DO asset management, the app won’t do it for you”.
There are some great technical solutions to some of the challenges of asset management, including discovery, categorisation and management, but like most things you need to make sure people and process are working otherwise the technology can’t help you. Your people need to understand the importance of accurate asset inventory, and you should do everything you can to manage shadow IT effectively. Your processes (e.g. change management) need to deliver the right outputs, and you need to have some level of assurance that your asset inventory is complete. For example, you could;
- Create a Security Information and Event Management (SIEM) alert on any successful authentication from a device which isn’t in your asset inventory,
- Run the same passive information gathering tools cyber-criminals and security researchers will use to understand your attack surface (shodan.io, crt.sh),
- Run discovery tools like Nmap to map your internal network and spot those forgotten IoT devices, unmanaged Wireless Access Points and Access Control systems stuck in a cupboard.
So try and ignore the somewhat tedious nature of asset management and make it your friend, because if you know what you’ve got, you’ll be much more able to protect it.
- Run this search on Shodan.io and look for a NetScaler system with an SSL certificate Last-Modified date before October 10th ↩︎