Listen to this Post
Introduction:
A crashing production server can bring a business to its knees, often due to mundane yet critical system administration failures. By mastering a core set of Linux commands, IT professionals can rapidly diagnose and resolve common infrastructure emergencies, transforming from panicked responders to calm problem-solvers.
Learning Objectives:
- Diagnose disk space and system resource issues using essential command-line tools.
- Analyze system logs to identify the root cause of application and service failures.
- Safely manage files and processes to restore system stability without causing further damage.
You Should Know:
1. Diagnosing Disk Space Emergencies
The first step in many server crises is checking disk utilization. A full root partition (/) will cause system instability, application crashes, and unresponsive services.
`du -sh /`
This command estimates file space usage. The flags `-s` summarizes the total for each argument, and `-h` prints sizes in human-readable format (e.g., 1K, 234M, 2G). Running it on the root directory (/) provides a immediate, high-level overview of which top-level directories are consuming the most space. A `/var` directory showing 99% usage is a classic culprit, often due to unchecked log file growth.
Step-by-Step Guide:
1. Connect to the affected server via SSH.
- Run `df -h` to confirm the root partition is at or near 100% capacity.
- Execute `du -sh /` to see which top-level directory is the largest.
- Navigate into the suspect directory (e.g.,
cd /var) and repeat `du -sh ` to drill down further. - Continue this process until you locate the specific large file or directory, commonly found in `/var/log` or
/tmp.
2. Real-Time Log File Investigation
When an application fails, system and application logs are the primary source of truth. Knowing how to quickly interrogate them is a fundamental skill.
`tail -n 50 /var/log/syslog`
This command displays the last 50 lines of the system log file. The `-n` flag specifies the number of lines from the end of the file to print. This is invaluable for seeing the most recent error messages or events that occurred just before a service crashed.
`tail -f /var/log/nginx/error.log`
The `-f` (follow) flag is critical for live debugging. It outputs appended data as the file grows. After restarting a crashed service, you can `tail -f` its log file to see real-time messages, confirming a successful start or immediately capturing new errors.
Step-by-Step Guide:
- Use `tail -n 100 /var/log/syslog` to get a snapshot of recent system-wide events.
- For application-specific issues, identify the correct log (e.g.,
/var/log/nginx/error.log,/var/log/mysql/error.log). - Use `tail -f [bash]` to monitor the log in real-time while you attempt to restart the problematic service.
- Look for repeating error patterns, permission denials, or connection timeouts.
3. Safely Managing Problematic Files
While the post humorously mentions rm -rf, this command is incredibly dangerous. Safer alternatives and rigorous habits are required to prevent catastrophic data loss.
`rm -i large_logfile.log`
The `-i` flag prompts for confirmation before every removal. This is a critical safety measure, especially when logged in as root.
`shred -zuvn 5 confidential_file.txt`
For secure deletion, `shred` overwrites a file to hide its contents. Flags: `-z` adds a final overwrite with zeros to hide shredding, `-u` removes the file after overwriting, `-v` shows verbose output, and `-n 5` specifies the number of overwrite passes.
`find /var/log -name “.log” -mtime +30 -exec rm -i {} \;`
This is a safer, targeted cleanup command. It uses `find` to locate files in `/var/log` ending in `.log` that were modified more than 30 days ago (-mtime +30) and interactively deletes them (-exec rm -i {} \;).
Step-by-Step Guide:
- Never run `rm -rf /` or `rm -rf ` without absolute certainty of your current directory and the files you are targeting.
- Before deleting, create a backup:
tar -czf log_backup.tar.gz /path/to/old/logs/. - Use `find` with `-exec` or `xargs` for precise, automated cleanup of old files, always testing with `-ls` first (e.g.,
find ... -ls). - Prefer log rotation (via
logrotate) over manual deletion to manage log file growth systematically.
4. Process and Resource Monitoring
A server can be unresponsive due to runaway processes consuming 100% of CPU or memory, not just disk space.
`top` / `htop`
The `top` command provides a dynamic, real-time view of running processes, sorted by CPU usage by default. `htop` is an enhanced, more user-friendly version.
`ps aux –sort=-%mem | head -10`
This `ps` command lists all processes (aux) and sorts them by memory usage in descending order (--sort=-%mem), then displays only the top 10 memory-hungry processes.
`lsof /var`
The `lsof` (list open files) command can reveal which processes are holding files open in a specific directory. If a partition is full, `lsof +L1` can help find deleted files that are still held open by processes, preventing the space from being freed.
Step-by-Step Guide:
- Run
top. Check the “load average” and “%Cpu(s)” at the top of the output. - Press `M` to sort processes by memory usage. Identify any consuming excessive resources.
- Use `ps aux | grep [bash]` to get more details on a specific suspect process.
- To terminate a process, first try
kill [bash]. If it does not respond, use `kill -9 [bash]` as a last resort.
5. Network and Service Connectivity Checks
Often, the problem is not the server itself but a network issue or a dependent service being down.
`ss -tuln`
This modern replacement for `netstat` shows all listening sockets. Flags: `-t` (TCP), `-u` (UDP), `-l` (listening), `-n` (numeric, don’t resolve service names). It instantly shows which services are running and on which ports.
`systemctl status nginx`
The `systemctl` command is used to control systemd services. `status` shows whether a service (e.g., nginx, mysql) is active (running), failed, or inactive, along with its most recent log entries.
`curl -I http://localhost:80`
This command fetches the HTTP headers from a web server. A return of `HTTP/1.1 200 OK` indicates the web service is responsive. It can also reveal redirects (301/302) or server errors (5xx).
Step-by-Step Guide:
- Use `ss -tuln | grep :80` to confirm a service is listening on the expected port (e.g., 80 for HTTP).
- Run `systemctl status [bash]` to check the service’s state.
- If the service is
inactive (dead), start it withsystemctl start [bash]. - If the service is
failed, use `journalctl -u [bash]` to get detailed logs for diagnosis. - Use `curl` or `wget` locally to test if the service is responding to requests.
What Undercode Say:
- Proactive Monitoring Trumps Reactive Panic. The “war room” scenario is a failure of monitoring, not just engineering. Implementing disk space alerts (e.g., with Prometheus, Datadog) at 80% capacity provides a buffer for orderly cleanup long before a crisis.
- The Terminal is a Scalpel, Not a Sledgehammer. The reflexive caution around `rm -rf` underscores that command-line power requires discipline. Building muscle memory for safe commands and establishing pre-deletion checklists (e.g., “Am I in the right directory? Did I make a backup?”) is non-negotiable for professional system administration.
The core lesson from this incident is that modern IT infrastructure, while complex, often fails in predictable ways. The skill gap isn’t in knowing the most obscure commands, but in having the foundational expertise to systematically apply a few dozen essential tools to quickly triage and resolve the 95% of common problems related to resources, configuration, and services. This transforms IT from a cost center constantly fire-fighting into a reliable, strategic function.
Prediction:
The increasing abstraction of infrastructure through containers and serverless platforms will shift, but not eliminate, these core systems skills. While an operator may not `SSH` into a specific container, the underlying principles of resource constraints (CPU, memory, ephemeral storage), log aggregation, and process lifecycles will remain critical. Future outages will be diagnosed by querying centralized log streams and metric databases with SQL-like languages or dedicated query languages (e.g., PromQL), but the logical process of deduction—moving from symptom (the website is down) to root cause (a misconfigured cloud credential rotated 12 hours ago)—will be the enduring and valuable skill. The tools will evolve, but the systematic, command-line-driven mindset of the sysadmin will become more valuable, not less.
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Gershon Avital – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
