Discussions around security solutions often focus on the threats they can detect and the breaches they can prevent. But what happens when an attacker gains privileged access to the system in which they operate? Though uncomfortable, this is a pertinent question. Powerful, new exploitation methods continue to be developed, such as SLUBStick, which can bypass existing security mitigations and achieve privilege escalation undetected. This raises an even bigger concern: how well do threat detection and monitoring tools fare post-compromise? Their prospects are bleak if, like most endpoint detection and response (EDR) or runtime security tools, they operate within the compromised environment, which is the very same one they are supposed to protect. Because that means a privileged adversary not only knows they exist, but can directly target and disable them.
In this article, we’ll demonstrate exactly how, on a Linux host provisioned with an up-to-date version of a well-known runtime security monitoring tool, an adversary with local access and a working privilege escalation exploit can disable defences to access sensitive system files while remaining undetected. We’ll walk you through the adversary’s subsequent stealthy deployment of a kernel-mode rootkit – dubbed rogue_mod developed by our own team and inspired from Diamorphine –, and show you how the attacker can tamper with log files to deprive security response teams of a usable forensic trail.
The attack shown here was conducted in an isolated environment, using well-known tactics and publicly disclosed vulnerabilities and exploits, and no real-world systems were affected. The videos below show the output of the adversary’s terminal (left of screen, in pink) and the system’s security logs (right of screen, in black) observed on our demo box during the various stages of the reproducible attack.
Inside the attack: demo videos and technical walkthrough
Stage 1: Gain root access and neutralise the security solution to access a secret password file without being detected
Stage 1a: Exploit a kernel vulnerability for privilege escalation
The adversary starts out with an unprivileged local account and the intention of accessing the Linux password file /etc/shadow. An initial attempt at access fails due to insufficient privileges. The attacker then runs a local privilege escalation exploit (against CVE-2023-2163 in the kernel’s eBPF subsystem) and gains superuser privileges. Although the vulnerability had been publicly known for over a year at the time of our test, this exploit was undetected by the then-current version of the installed security tool. It's quite possible this omission was more than a simple matter of threat definitions not being kept up to date: exploits based on low-level kernel vulnerabilities can be hard if not impossible to detect, and likewise to precisely characterise.
With superuser privileges, the attacker is now able to access /etc/shadow. For purposes of demonstration, we have our attacker do this. The access to the sensitive file is detected by the security solution, which raises an alert (the logs show a notification “sensitive file opened for reading by non-trusted program”). But what if the attacker proceeded more carefully, by first disabling the security solution?
Stage 1b: Disable kernel event monitoring by the security tool
The privileged attacker can readily achieve the disabling of the security solution. The most obvious way to do this would be to stop the security solution completely, but that would be very visible, so the attacker takes a subtler approach by interfering with the security solution’s collection of kernel events. Like most EDR tools, the installed security solution detects threats by monitoring this source of data, which in its case is provided by eBPF filters. By deleting keys in eBPF maps, the script disable_edr.sh stops communication between the kernel and the eBPF filters so that the solution becomes effectively blind while appearing to be working as per normal. This act of disabling event collection itself goes undetected, which is unsurprising because it’s done using perfectly legal eBPF instructions, and it’s not possible in general to determine whether any given modification of eBPF maps was done maliciously.
After communication with the eBPF filters has been tampered with, the security solution has no way of detecting access to the sensitive file, and there is no log activity when /etc/shadow is opened, showing that the adversary can achieve its objective of covertly reading the system password file. Of course, the runtime security tool could attempt to restore its eBPF maps, but it would already be too late, and in any case the adversary could simply run disable_edr.sh again. We have ended up in this cat-and-mouse situation because the security tool’s event collection operates from the same environment as the adversary with superuser privileges. The outcome would be the same even if the security tool used a different method of event monitoring, such as system call hooking, since with root privileges the attacker could similarly disable that.
Stage 2: Install malicious kernel module that implements a rootkit
The module’s presence is hidden from the system’s list of installed modules.
In the next stage of the attack, the adversary aims to install and hide a malicious Linux kernel module (rogue_mod.ko). Because the eBPF filters were disabled in the previous stage, the security solution doesn’t detect the installation of the kernel module (via the command insmod rogue_mod.ko). Once installed, the kernel module implements a kernel-mode rootkit, modifying kernel code to conceal its presence from the rest of the system: it’s seen that the module rogue_mod doesn’t appear in the results of the lsmod command, which is supposed to list all installed kernel modules. However, the module really is present: when the adversary pauses the rootkit by issuing the command unhide_rogue_module.sh we see that rogue_mod is shown in the output of lsmod.
This detection failure is particularly serious, because hiding its own presence is the tip of the iceberg of rogue_mod’s potential capabilities: a kernel-mode rootkit can modify any part of the kernel’s code, and could selectively hide filesystem operations, conceal the existence of malicious processes, and even tamper with the communication channel between the EDR tool and the central servers it reports suspicious events to – just to name a few actions in the domain of defence evasion.
Stage 3: Delete command history to cover their tracks
We have seen how the adversary can fully compromise the system and, if careful, remain completely undetected by the runtime security monitoring tool while doing so. Having achieved its objectives, the adversary would now like to cover their tracks so as to leave no trace of their actions for potential forensics investigations.
The deletion is detected but the command history has been lost.
The Bash command shell on Linux keeps a log of issued commands in the file .bash_history located in a user’s home directory. When the attacker deletes this file, the runtime security tool raises a notification “Bash history deleted”, which corresponds to the MITRE technique “T1070.003 – Indicator Removal: Clear Command History”. This is good, but for the security team responding to the alert, it also indicates that the record of the adversary’s actions has been lost. A key component of the forensic trail is gone forever, and this will hinder efforts towards making a targeted recovery from the attack.
While the example might suggest the immediate remedy of preserving security logs somewhere beyond the reach of attackers, this would miss a more fundamental observation: the very fact that the task of collection and emission of security artifacts is performed from inside the system being monitored already means that it is at risk of being interfered with. The attacker in this example could have used its rootkit to filter communications between the endpoint security solution and an immutable database, so that its command history never gets permanently recorded.
And that concludes the walkthrough of our demonstration. As we've seen, an attacker with root access can disable the installed endpoint security tools while remaining undetected. If security solutions exist within the environment they protect, they become vulnerable themselves, leaving defenders at a disadvantage. So how can they break out of this cycle?
Ending the cat-and-mouse game by shifting threat detection to a different level
The example attack just described highlights the fragility of a security strategy that relies substantially on agent-based security monitoring and threat detection. Once an attacker gains superuser privileges and can compromise its kernel at will, no part of the system is safe. Any security solution located within the compromised system is at risk. And because no security solution is exhaustive, a skilful adversary will likely find ways to disable it or tamper with it without being detected. This creates a never-ending cat-and-mouse game between attackers and defenders – one that traditional endpoint security tools are unlikely to win.
At Ryzome, we believe there is a way out of this stalemate, at least in virtualised environments: organisations can shift security monitoring and threat detection to the hypervisor level. This approach keeps security tools out of reach of attackers while maintaining continuous, unimpaired visibility into endpoints. Depending on the nature of your workloads, this shift could provide a more resilient and effective security posture.
If you’re interested in learning more, contact us.
