CVE-2023-3519 CVE-2022-27513 and CVE-2023-3466
This vulnerability is a flashback to 2019 that shook the Citrix world with CVE-2019-19781. This allowed the ability to execute code remotely by allowing the upload of code to insecure directories, along with some other contributing factors. In 2019 Citrix unintentionally released enough information within the disclosure with how to defend\mitigate against exploitation before you could patch. That information was weaponized quickly band allowed security researchers and others to make exploits that could use the vulnerability.
Fast forward to 2023, and the main Remote Code Execution CVE in this notification, CVE-2023-3519, has one line that worries me the most “Exploits of CVE-2023-3519 on unmitigated appliances have been observed.” I haven’t been able to find a public exploit to test and validate, but this means that Citrix has found indicators of compromise on systems. They have not released how they know things were compromised, but if we look at the same things we used back in 2019, we may be close to understanding possible persistence paths.
The one thing to think about any new CVE is that it typically will just give that first initial execution path, but what an attacker does after that will vary. So, in many cases, there isn’t a definite next step, and the attacker’s creativity can help them hide in the shadows of the system and possibly create persistence. There is no sure thing about cyber security, and we must be able to adapt and be ready to react.
Files must most likely be placed under some public /var based directories that the Citrix Gateway and its supporting systems use, where many of the problems were back in 2019.
Here is a little List of things to do (Disclaimer: This isn’t guaranteed to catch everything, but these are just good starting points)
1. Find out when you last updated your Citrix NetScaler
a. shell ls -ll /var/nsinstall
b. Convert to YYYYMMDD based on the date of the install and add an extra day.
c. If you have customized your Citrix Gateway page or layout files you should see those edits that may have been done after that date, and that can be expected.
2. After using the find command, look at a couple of directories to see if there are things newer than that date.
a. find /netscaler/ns_gui/ -type f -name *.php -newermt { Timestamp of Installer Files Date +1} -exec ls -l {} \;
b. find /var/vpn/ -type f -newermt { Timestamp of Installer Files Date +1} -exec ls -l {} \;
c. find /var/netscaler/logon/ -type f -newermt { Timestamp of Installer Files Date +1} -exec ls -l {} \;
d. find /var/python/ -type f -newermt {Timestamp of Installer Files Date +1} -exec ls -l {} \;
e. find /var/ -type f -newermt {Timestamp of Installer Files Date +1} -exec ls -l {} \;
3. Log hunting is important too. This is where you will want to look for Shell files and PHP files that could be getting queried\executed on your system. Many of these will be system-based files.
a. zgrep ‘\.sh’ /var/log/httperror.log*
b. zgrep ‘\.pl’ /var/log/httperror.log*
c. zgrep ‘\.php’ /var/log/httperror.log*
d. grep ‘/flash/nsconfig/keys’ /var/log/sh.log*
i. What commands have been executed on the system?
e. grep -v ‘127\.0\.0’ /var/log/*.log | grep ‘nc -l\|/etc/passwd\|/etc/shadow\|python -c\|curl\|\.php’
i. Looking for things related to configuration files, passwords, and the execution of Python and curl.
4. Look for edited files with the setuid bit
a. find /var -perm -4000 -user root -not -path “/var/nslog/*” -newermt {Timestamp of Installer Files Date +1} -exec ls -l {} \;
5. Find Nobody processes, this could be filtered down to specific paths, most likely under /var but this should give you some clues to chase down.
a. shell ps aux | grep nobody
6. This list is not everything possible where execution or logs would be detectable but it’s a great start. I know that Citrix has some IoCs that are available via support that I think will keep growing.
Another important note is that during a time like this, many scripts and commands will come out, and you need to be careful about what you’re running.
What to do now?
1. Check for IoCs if possible, using some of the scripts above or from other sources. I would guess Citrix may release them soon after they continue to collect them and as time passes for more customers to patch. Remember IoCs from anyone are not perfect because if someone changes the attack avenue to use different methods, file types or many other variables.
1. If you find suspicious things, you can reach out to your Citrix Partner or Citrix Support to help escalate and validate if those findings are valid.
2. If the attack is validated, then you need to work on changing the “nsroot” password and all service accounts especially bind accounts. Depending on the threat profile, you may need to bring the host offline and reimage the device once you have collected enough forensic information. A firmware upgrade “should” wipe persistence and the connection to each appliance unless there have been some more advanced techniques used. You may also want to rekey SSL certificates too.
2. Patch and Patch. This isn’t one you want to wait around for. The longer you’re not patched, the higher your risk will be for exploitation and pivots into other systems.
3. After patching, continue to monitor your system to ensure there isn’t any unusual activity.
4. Ensure your device is hardened (Top 3)
1. Your NSIP should only be accessible from management machines and networks. If you can sit at a random client machine and access the management of the NetScaler, you’re not protecting your deployment effectively.
2. Bind to LDAP. Never log in with “nsroot” to do work. Ensure the password is strong and change it yearly along with the Bind account also. Ensure your bind account is just a Domain User and nothing more; for many years, people added this account to privileged groups that aren’t required.
3. Ensure you have syslog configured. You need the logs off the box for notifications and forensics. This is where it’s recommended that you have SIEM to aggregate this information.
Looking around:
SHODAN shows around ~88k NetScalers for Gateway, VPN, or RDP Proxy. This total isn’t counting just vulnerable systems, as that scan hasn’t been created or released yet. This is a similar amount to the 2019 incident. There could be more appliances if some default items were changed or missed during the SHODAN’s regular scanning intervals or have opted out of their scans for their domain or IP space. https://www.shodan.io/search?query=http.title%3A%22Citrix+Login%22
This is a combined look at both main page titles. With so many appliances online, many of these could be honeypots to help gain other methods and techniques. There weren’t this many appliances in China in 2019 as a percentage.
https://www.shodan.io/search?query=http.title%3A%22Citrix+Login%22%2C%22Netscaler%22
2019 Shodan Output for Reference
Relevant Links
Great Indicators of Compromise from the US-based CISA. This is a must-read article because it outlines what they have had reported to them, and it points to attackers heading straight to Active Directory to do things from there. The attackers also c
Great Checklist Manuel Winkel as always
https://www.deyda.net/index.php/en/2023/07/19/checklist-for-citrix-adc-cve-2023-3519/
This is a great look at reverse engineering the previous version and the patched version to find the deltas and then looking at some of the possible attack avenues from there. Great Work rbowes-r7! (Conclusion: Possibly SAML related)
https://attackerkb.com/topics/si09VNJhHh/cve-2023-3519
Secure Deployment Guide for Citrix NetScaler (NetScaler Hardening)
https://docs.netscaler.com/en-us/citrix-adc-secure-deployment.html
Just checks for the Firmware version to see if it’s lower than the recommended versions. https://github.com/telekom-security/cve-2023-3519-citrix-scanner/blob/main/CVE-2023-3519-checker.py
Reverse Engineering Patch Notes (Conclusion: Possibly SAML related)
https://blog.assetnote.io/2023/07/21/citrix-CVE-2023-3519-analysis/
Some relevant Tweets about this situation.
https://twitter.com/uk_daniel_card/status/1682281596898930689?s=46&t=XWft99MkHnHYiOPyo8ajNw
Good Stats that are scan based outside of Shodan
https://twitter.com/Shadowserver/status/1682355280317919233
Talking about IoCs
https://twitter.com/cyb3rops/status/1681591644498305024
Conclusion
This is a bad one, will it be as bad as 2019, to be determined? I think Citrix has learned a lot from the last major RCE from 2019 and has implemented some things I think they did much better this time. They have notifications when you login into Citrix.com, check your appliances via ADM, and even into the console itself. There were many waves of emails going out with some great notifications. They have some more detail but some less in their notification to help slow the exploits, but since they mentioned spotting exploited systems, we know there is someone with it. They haven’t shared many details on the exploit or indicators of compromise, as I think they want more people to patch before they do. I see both sides of this story, and I don’t know the perfect answer other than knowledge is power, and it’s also part of trust when it’s shared. Read the CISA report referenced above, as it gives you many details of the next steps after exploitation. Keep your ears and eyes open for more information, and I hope this helps someone. I will update this article as more information becomes available.
Timeline
1. July 18th, 2023, a Vulnerability notice came out from Citrix for their Citrix NetScaler product line. It has three but one has RCE capabilities. It was noted that exploits had been witnessed.
