Sandfly 5.2 - Linux Stealth Rootkit File and Directory De-Cloaking
S
Steve Busko
started a topic
about 1 month ago
Sandfly 5.2 has a powerful new way to detect Linux stealth rootkits: Hidden file and directory de-cloaking.
This feature will make files and directories hidden by many types of stealth rootkits immediately visible. More importantly, many of the ways rootkits hide their files and directories from detection on Linux will no longer work. We turn hiding from an asset into a liability.
We encourage customers to upgrade and report back any findings. Please read more about this important new feature below.
Linux Stealth Rootkit Background and Tactics
There are many types of rootkits available on Linux from very simple to advanced. At the top end of the threat spectrum are what is called Loadable Kernel Module (LKM) or simply stealth rootkits. These rootkits are characterized by some common traits:
They intercept kernel system calls to hide their presence.
They hide processes.
They hide network activity.
They hide directories.
They hide files and the data they contain.
Sandfly has had mechanisms to find variants of these tactics for some time, but we wanted to go a step further. In particular, we wanted to detect and de-cloak files or directories hidden by these rootkits and make the entire hiding tactic no longer work for any of them. And, we wanted to do it with our proven fast and safe agentless mechanism.
We're happy to announce that we've succeeded. Sandfly 5.2 makes the entire class of hiding files and directories with traditional stealth rootkit methods easily spotted.
Stealth Rootkit Hiding Methods
Linux stealth rootkits hook various system calls to hide processes, files, directories and even data inside files they do not want seen. Once these mechanisms are active, administrators and security teams logging into a host will see no signs of intruder activity.
The illustration above shows the basic flow of a stealth rootkit on Linux. Attackers will hook relevant system calls that display running processes, directories, files, network activity and even data they don't want revealed in files (e.g. startup commands for persistence). Once hooked, the system calls are run through the rootkit and data is modified to appear innocent before being returned.
Turning an Advantage into a Liability
What if we could see what files and directories a rootkit is hiding? If we could do this, then what is normally an advantage for the rootkit (being able to hide) actually turns into a significant disadvantage because it instantly reveals the system is compromised. This is what Sandfly 5.2 and our file and directory de-cloaking does: We make hiding on Linux a liability.
Let's see this liability in action. We'll use two common stealth rootkits: Diamorphine and Reptile. These rootkits are used as a base for many variants. They both use kernel hooking methods to hide files and directories that match secret words (e.g. the file or directory has the word "reptile" or "diamorphine" in it).
Below we flag a hidden directory from Diamorphine under the system /bin directory. It is immediately obvious something is hiding from view.
Next we look at a system running Reptile and identify the startup file being used to maintain persistence using udev rules. Notice that the file is not only de-cloaked, but all file forensic attributes are available including creation dates, owner, cryptographic hashes, and more. This happens even if you can't see the file on the host with other tools.
Most importantly, this is a high-fidelity alert. The rootkit actually calls out like a blaring siren that something is wrong by hiding.
De-Cloaking Modules in Sandfly 5.2
Sandfly checks high-risk areas for signs of cloaked files or directories. The modules below are enabled by default with the exception of the anywhere check, which is designed for use during incident response to scan the entire file system if needed.
dirs_cloaked_entry_bin - Cloaked files and directories under common system binary locations.
dirs_cloaked_entry_etc - Cloaked files and directories under /etc.
dirs_cloaked_entry_lib - Cloaked files and directories under library and kernel module locations.
dirs_cloaked_entry_root - Cloaked files and directories under the system / (root) locations.
dirs_cloaked_entry_system - Cloaked files and directories under system areas such as /boot.
dirs_cloaked_entry_usr_* - Cloaked files and directories under various /usr locations.
dirs_cloaked_entry_var - Cloaked files or directories under /var locations.
dirs_cloaked_entry_anywhere - Cloaked files and directories anywhere on the disk (incident response check).
Customers can clone any of these and make custom directories they wish to check on demand.
Stealth Rootkit Detection Arsenal To Date
In addition to these new features, all of the following stealth rootkit detection methods are in our core product for immediate deployment agentlessly.
De-cloaking hidden processes - Finds binaries running with a hidden process ID (PID).
Detecting kernel taint inconsistencies - Finds tainted kernels that have had unsigned or unknown modules loaded but are hiding from view.
Directory link count inconsistencies - Finds directories that have one or more entries hidden due to inconsistent link counts (e.g. kernel says 3 directories but file system reports 4 present).
File size mismatches - Finds files hiding data due to inconsistent byte counts vs. reported file size (e.g. kernel says file is 100 bytes, but file system says it is 120 bytes).
Suspicious activity with LD_PRELOAD - Finds user space rootkits using LD_PRELOAD as attack vector. This is not kernel space rootkit, but we include it here for completeness.
File and directory de-cloaking - Our new ability here to instantly reveal files or directories being actively hidden by a stealth rootkit on Linux.
Technical Considerations
De-cloaking works on EXT4 and XFS file systems. We may add support for other file systems in the future. However, these two file systems are the majority of Linux installs globally and we expect extremely wide coverage for this threat. The capability will work across any architecture Sandfly supports such as Intel, AMD, ARM, MIPS and IBM POWER CPUs. It also will work on servers, appliances and even embedded devices. As with all Sandfly installs, it also works on systems up to a decade+ old to modern distributions, even those in the cloud or on-premise.
Keeping with our core philosophy, these methods do not tie into kernel space and they work agentlessly. This significantly decreases system performance and stability risks commonly associated with agent-based approaches. Sandfly is compatible, fast and safe.
Help Us Find New Rootkits
We believe this new de-cloaking feature is likely to find previously unknown rootkits operating in the wild. If you find any files uncovered by this new feature, please reach out to our support team. We would like to see any binaries or directories uncovered by this method to help other Linux security teams. We will handle any contacts confidentially.
Other New Detection Modules
In addition to file and directory de-cloaking, we have also added the following:
Collecting volume used drive space - Operating System JSON data now collects drive space used by each mount point as the used_percent value.
policy_disk_any_file_system_full - Policy that can be enabled to alert if a volume is more than 95% full on a host.
last_scan_duration_seconds is available in the host JSON header data. This will track how long the scan took in total seconds to allow teams to better track performance of scans.
file_etc_xdg_autostart_exec_base64 - Finds /etc/xdg files with base64 encoding often done to obfuscate malicious persistence commands.
file_etc_xdg_autostart_exec_hex - Finds /etc/xdg files with hex encoding often done to obfuscate malicious persistence commands.
file_etc_xdg_autostart_exec_malicious - Finds /etc/xdg command that have an appearance of being malicious or suspicious.
file_etc_xdg_autostart_exec_command_hidden - Finds /etc/xdg with paths calling to hidden commands.
file_etc_xdg_autostart_exec_directory_hidden - Finds /etc/xdg files calling into hidden directories.
file_etc_xdg_autostart_exec_from_shm_dir - Finds /etc/xdg files calling to system /dev/shm ramdisk locations.
file_etc_xdg_autostart_exec_from_tmp_dir - Finds /etc/xdg files calling into system tmp directories.
policy_file_permissions_risk_etc_xdg_autostart - Finds /etc/xdg scripts with risky or bad permissions set (e.g. world writable).
policy_user_ssh_authorized_keys_type_dss - Optional policy check to flag users with deprecated SSH DSS key types.
recon_file_attributes_in_udev_dir - Pulls files for recon and drift detection tracking under /etc/udev and /lib/udev directories.
We have corrected false positives happening for certain Kubernetes installs during container scanning as well.
Get a Free License Today
If you have not tried Sandfly, get your free license below:
Steve Busko
Sandfly 5.2 has a powerful new way to detect Linux stealth rootkits: Hidden file and directory de-cloaking.
This feature will make files and directories hidden by many types of stealth rootkits immediately visible. More importantly, many of the ways rootkits hide their files and directories from detection on Linux will no longer work. We turn hiding from an asset into a liability.
We encourage customers to upgrade and report back any findings. Please read more about this important new feature below.
Linux Stealth Rootkit Background and Tactics
There are many types of rootkits available on Linux from very simple to advanced. At the top end of the threat spectrum are what is called Loadable Kernel Module (LKM) or simply stealth rootkits. These rootkits are characterized by some common traits:
They intercept kernel system calls to hide their presence.
They hide processes.
They hide network activity.
They hide directories.
They hide files and the data they contain.
Sandfly has had mechanisms to find variants of these tactics for some time, but we wanted to go a step further. In particular, we wanted to detect and de-cloak files or directories hidden by these rootkits and make the entire hiding tactic no longer work for any of them. And, we wanted to do it with our proven fast and safe agentless mechanism.
We're happy to announce that we've succeeded. Sandfly 5.2 makes the entire class of hiding files and directories with traditional stealth rootkit methods easily spotted.
Stealth Rootkit Hiding Methods
Linux stealth rootkits hook various system calls to hide processes, files, directories and even data inside files they do not want seen. Once these mechanisms are active, administrators and security teams logging into a host will see no signs of intruder activity.
The illustration above shows the basic flow of a stealth rootkit on Linux. Attackers will hook relevant system calls that display running processes, directories, files, network activity and even data they don't want revealed in files (e.g. startup commands for persistence). Once hooked, the system calls are run through the rootkit and data is modified to appear innocent before being returned.
Turning an Advantage into a Liability
What if we could see what files and directories a rootkit is hiding? If we could do this, then what is normally an advantage for the rootkit (being able to hide) actually turns into a significant disadvantage because it instantly reveals the system is compromised. This is what Sandfly 5.2 and our file and directory de-cloaking does: We make hiding on Linux a liability.
Let's see this liability in action. We'll use two common stealth rootkits: Diamorphine and Reptile. These rootkits are used as a base for many variants. They both use kernel hooking methods to hide files and directories that match secret words (e.g. the file or directory has the word "reptile" or "diamorphine" in it).
Below we flag a hidden directory from Diamorphine under the system /bin directory. It is immediately obvious something is hiding from view.
Next we look at a system running Reptile and identify the startup file being used to maintain persistence using udev rules. Notice that the file is not only de-cloaked, but all file forensic attributes are available including creation dates, owner, cryptographic hashes, and more. This happens even if you can't see the file on the host with other tools.
Most importantly, this is a high-fidelity alert. The rootkit actually calls out like a blaring siren that something is wrong by hiding.
De-Cloaking Modules in Sandfly 5.2
Sandfly checks high-risk areas for signs of cloaked files or directories. The modules below are enabled by default with the exception of the anywhere check, which is designed for use during incident response to scan the entire file system if needed.
dirs_cloaked_entry_bin - Cloaked files and directories under common system binary locations.
dirs_cloaked_entry_etc - Cloaked files and directories under /etc.
dirs_cloaked_entry_lib - Cloaked files and directories under library and kernel module locations.
dirs_cloaked_entry_root - Cloaked files and directories under the system / (root) locations.
dirs_cloaked_entry_system - Cloaked files and directories under system areas such as /boot.
dirs_cloaked_entry_usr_* - Cloaked files and directories under various /usr locations.
dirs_cloaked_entry_var - Cloaked files or directories under /var locations.
dirs_cloaked_entry_anywhere - Cloaked files and directories anywhere on the disk (incident response check).
Customers can clone any of these and make custom directories they wish to check on demand.
Stealth Rootkit Detection Arsenal To Date
In addition to these new features, all of the following stealth rootkit detection methods are in our core product for immediate deployment agentlessly.
De-cloaking hidden processes - Finds binaries running with a hidden process ID (PID).
Reveal hidden kernel modules - Detects kernel modules hiding their presence.
Detecting kernel taint inconsistencies - Finds tainted kernels that have had unsigned or unknown modules loaded but are hiding from view.
Directory link count inconsistencies - Finds directories that have one or more entries hidden due to inconsistent link counts (e.g. kernel says 3 directories but file system reports 4 present).
File size mismatches - Finds files hiding data due to inconsistent byte counts vs. reported file size (e.g. kernel says file is 100 bytes, but file system says it is 120 bytes).
Suspicious activity with LD_PRELOAD - Finds user space rootkits using LD_PRELOAD as attack vector. This is not kernel space rootkit, but we include it here for completeness.
File and directory de-cloaking - Our new ability here to instantly reveal files or directories being actively hidden by a stealth rootkit on Linux.
Technical Considerations
De-cloaking works on EXT4 and XFS file systems. We may add support for other file systems in the future. However, these two file systems are the majority of Linux installs globally and we expect extremely wide coverage for this threat. The capability will work across any architecture Sandfly supports such as Intel, AMD, ARM, MIPS and IBM POWER CPUs. It also will work on servers, appliances and even embedded devices. As with all Sandfly installs, it also works on systems up to a decade+ old to modern distributions, even those in the cloud or on-premise.
Keeping with our core philosophy, these methods do not tie into kernel space and they work agentlessly. This significantly decreases system performance and stability risks commonly associated with agent-based approaches. Sandfly is compatible, fast and safe.
Help Us Find New Rootkits
We believe this new de-cloaking feature is likely to find previously unknown rootkits operating in the wild. If you find any files uncovered by this new feature, please reach out to our support team. We would like to see any binaries or directories uncovered by this method to help other Linux security teams. We will handle any contacts confidentially.
Other New Detection Modules
In addition to file and directory de-cloaking, we have also added the following:
Collecting volume used drive space - Operating System JSON data now collects drive space used by each mount point as the used_percent value.
policy_disk_any_file_system_full - Policy that can be enabled to alert if a volume is more than 95% full on a host.
last_scan_duration_seconds is available in the host JSON header data. This will track how long the scan took in total seconds to allow teams to better track performance of scans.
file_etc_xdg_autostart_exec_base64 - Finds /etc/xdg files with base64 encoding often done to obfuscate malicious persistence commands.
file_etc_xdg_autostart_exec_hex - Finds /etc/xdg files with hex encoding often done to obfuscate malicious persistence commands.
file_etc_xdg_autostart_exec_malicious - Finds /etc/xdg command that have an appearance of being malicious or suspicious.
file_etc_xdg_autostart_exec_command_hidden - Finds /etc/xdg with paths calling to hidden commands.
file_etc_xdg_autostart_exec_directory_hidden - Finds /etc/xdg files calling into hidden directories.
file_etc_xdg_autostart_exec_from_shm_dir - Finds /etc/xdg files calling to system /dev/shm ramdisk locations.
file_etc_xdg_autostart_exec_from_tmp_dir - Finds /etc/xdg files calling into system tmp directories.
policy_file_permissions_risk_etc_xdg_autostart - Finds /etc/xdg scripts with risky or bad permissions set (e.g. world writable).
policy_user_ssh_authorized_keys_type_dss - Optional policy check to flag users with deprecated SSH DSS key types.
recon_file_attributes_in_udev_dir - Pulls files for recon and drift detection tracking under /etc/udev and /lib/udev directories.
We have corrected false positives happening for certain Kubernetes installs during container scanning as well.
Get a Free License Today
If you have not tried Sandfly, get your free license below:
Get Sandfly
Upgrading Sandfly
All customers are encouraged to upgrade to check their systems for potential rootkit activity using these new features.
We are here to help with any questions. Please see our documentation on the new features and capabilities:
Sandfly Documentation
Customers wishing to upgrade can follow the instructions here:
Upgrading Sandfly
If you have any questions, please reach out to us.
Thank you for using Sandfly.