Listen to this Post
Introduction:
In the absence of EDR (Endpoint Detection and Response) or traditional antivirus solutions, a Linux system administrator must rely on the kernel’s internal telemetry to detect compromise. Attackers often install reverse shells or bind shells that listen for incoming connections. By using native “Living off the Land” binaries, defenders can query the system’s socket table to uncover hidden listeners. This article provides a practical, command-line approach to manual threat hunting on Linux, focusing on the `ss` command as a primary investigative tool.
Learning Objectives:
- Understand how to use the `ss` utility to audit listening ports.
- Learn to differentiate between legitimate services and malicious backdoors.
- Master process-to-port mapping to identify suspicious binaries.
- Explore complementary Linux commands for deep-dive investigations.
- Implement quick mitigation techniques for discovered threats.
You Should Know:
- The Power of
ss: Reading the Kernel’s Socket Table
Modern Linux systems no longer default tonetstat; instead, `ss` (socket statistics) is the tool of choice. When a backdoor is installed—whether it’s a Python one-liner, a Netcat listener, or a Meterpreter shell—it must open a port to communicate. The kernel maintains a real-time list of all sockets. By executing the following command, you force the system to reveal exactly what is listening and which process owns it:sudo ss -tulpn
Breakdown of the flags:
-t: Display TCP sockets.-u: Display UDP sockets.-l: Show only listening sockets (omit established connections).-p: Show the process using the socket (requires root).-n: Show numerical addresses and ports (avoid DNS lookups).
What to look for:
- Unusual ports: Any port above 1024 that isn’t a known service (e.g., 4444, 1337, 31337).
- Suspicious processes: Random Python scripts,
nc,bash,perl, or `ncat` listening. - Foreign IPs bound: If a service is listening on `0.0.0.0` (all interfaces) and it shouldn’t be, investigate immediately.
2. Mapping the Backdoor: From Port to Binary
Once you identify a suspicious listening port (e.g., 0.0.0.0:4444), the `-p` flag already gives you the PID and process name. However, for deeper analysis, extract the full path and environment:
ls -la /proc/<PID>/exe cat /proc/<PID>/cmdline | tr '\0' ' '
– The `/proc/
– `/proc/
If the process is malicious, you can immediately kill it:
sudo kill -9 <PID>
But do not stop there—backdoors often have persistence mechanisms.
3. Hunting Hidden Processes and Rootkits
Advanced adversaries might hide processes from `ps` and `ss` using loadable kernel modules (LKMs). In such cases, compare the output of `ss` with alternative tools:
sudo lsof -i -P -n | grep LISTEN
`lsof` lists open files, and sockets are considered files. If `ss` shows a process but `lsof` does not, or vice versa, it may indicate user-land hooking.
Additionally, check for unusual network connections using:
sudo netstat -tunap
(If `netstat` is installed, though `ss` is preferred.)
4. Investigating Suspicious Processes with `ps` and `htop`
Cross-reference the PID from `ss` with process lists to see CPU/memory usage and parent process:
ps aux | grep <PID> pstree -p <PID>
Look for:
- Processes running as root that shouldn’t be.
- Shells (
bash,sh,zsh) listening on network ports (extremely rare for legitimate services). - High CPU usage from a process named `kworker` or `systemd` (common masquerading names).
- Checking for Persistence: Cron, Systemd, and Startup Scripts
Attackers ensure their backdoor survives reboots. Common persistence locations include:
– Cron jobs: `crontab -l` and check /etc/crontab, /etc/cron.d/.
– Systemd services: `systemctl list-units –type=service –state=running` and look for recently added or suspiciously named services.
– Init scripts: /etc/rc.local, /etc/init.d/, and `~/.bashrc` for user-level persistence.
– SSH authorized_keys: Check `/root/.ssh/authorized_keys` and user home directories for unauthorized keys.
Use `grep` to search for reverse shell patterns in common persistence files:
grep -r "nc -e" /etc/cron /etc/systemd/system/
6. Memory Forensics: Dumping the Process
If you suspect the process is hiding its network activity, you can dump its memory for offline analysis:
sudo gcore <PID>
This creates a core dump file. You can then scan it for strings related to command-and-control (C2) IPs or configuration data:
strings core.<PID> | grep -E "([0-9]{1,3}.){3}[0-9]{1,3}"
7. Mitigation and Cleanup Steps
After confirming a backdoor:
- Isolate the machine: Block traffic at the firewall level immediately:
sudo iptables -A INPUT -s <attacker_IP> -j DROP sudo iptables -A OUTPUT -d <attacker_IP> -j DROP
2. Terminate the process: `kill -9 `.
- Remove persistence: Delete cron jobs, systemd service files, and malicious scripts.
- Rotate credentials: The attacker may have captured passwords; change all relevant secrets.
- Patch the entry point: Analyze logs to determine how the attacker gained access (e.g., vulnerable web app, weak SSH password).
What Undercode Say:
- Key Takeaway 1: Native Linux tools are powerful threat-hunting instruments. Commands like
ss,lsof, and `/proc` inspection often outperform commercial AV in detecting unknown malware because they query the kernel directly. - Key Takeaway 2: Persistence mechanisms are the attacker’s lifeline. Always check cron, systemd, and startup scripts after killing a backdoor; otherwise, the system will be re-infected within minutes.
- Analysis: The “Living off the Land” approach is a double-edged sword. While it allows defenders to hunt effectively without agents, it also means attackers use these same tools to blend in. The key differentiator is knowledge: understanding what “normal” looks like on your servers. Regular baselining of listening ports, processes, and user behavior is essential. In a world where supply chain attacks and zero-days evade signature-based detection, manual investigation skills remain the last line of defense.
Prediction:
As EDR evasion techniques become more sophisticated, we will see a resurgence in “manual” forensics. Attackers will increasingly target the `ss` and `lsof` binaries themselves (via LD_PRELOAD or binary patching) to hide their sockets. Future threats will require defenders to use statically compiled binaries or even raw `/proc` filesystem parsing from a trusted USB boot environment to avoid tampered system tools. The cat-and-mouse game between kernel-level rootkits and forensic analysts will intensify, making immutable infrastructure and ephemeral containers critical for post-compromise recovery.
▶️ Related Video (82% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Thantzinmoe Cybersecurity – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
