Nmap for Pentester: Decoding Port States to Bypass Firewalls Like a Pro

By /

Listen to this Post

Featured Image

Introduction:

Network reconnaissance is the bedrock of any penetration test, and Nmap’s ability to classify port states—beyond just “open” or “closed”—reveals how firewalls and services truly respond to probes. Understanding states like filtered, unfiltered, and open|filtered allows pentesters to infer security controls, identify attack surfaces, and craft evasion techniques that turn ambiguous scan results into actionable intelligence.

Learning Objectives:

  • Differentiate between the six Nmap port states (open, closed, filtered, unfiltered, open|filtered, closed|filtered) based on packet responses.
  • Perform SYN, ACK, and UDP scans to trigger each state in a controlled lab environment using iptables and Wireshark.
  • Apply mitigation strategies to control what an attacker learns from port scans, reducing reconnaissance exposure.

You Should Know:

  1. The TCP Handshake and How Nmap Derives Port States

Nmap’s port classification relies on the TCP three-way handshake and the responses (or lack thereof) from the target. When a SYN packet is sent:
– Open port replies with SYN/ACK → Nmap resets the connection.
– Closed port replies with RST/ACK → no service listening.
– Filtered → no response or ICMP unreachable (firewall dropped the probe).
– Unfiltered → reachable but state unknown (only detectable via ACK scan).
– open|filtered → ambiguous (common for UDP or no-response probes).
– closed|filtered → unable to differentiate closed from filtered.

Step‑by‑step guide to observe states using Kali Linux and a target VM:

  1. Set up two VMs (Kali attacker: 192.168.1.10, target Linux: 192.168.1.9). Ensure connectivity.
  2. On the target, check open ports: `sudo ss -tuln`
    3. From Kali, run a basic SYN scan: `sudo nmap -p80 192.168.1.9`
    4. Capture traffic on the target: `sudo tcpdump -i eth0 -1n port 80 -v`
    5. Observe the SYN → SYN/ACK → RST sequence for an open port.
  3. For a closed port, scan an unused port: sudo nmap -p9999 192.168.1.9. Wireshark shows SYN → RST/ACK.

Linux command to force a closed port response (no service listening):
– `sudo nmap -p9999 192.168.1.9` → expect `closed` state.

Windows alternative (using PowerShell and Test-1etConnection):

– `Test-1etConnection -Port 9999 -ComputerName 192.168.1.9` → shows `TcpTestSucceeded: False` (analogous to closed).

2. Manipulating Filtered Ports with iptables (Linux Firewall)

A filtered port occurs when a firewall drops packets without any reply or sends an ICMP “administratively prohibited” message. We can simulate this using iptables on the target.

Step‑by‑step to create a filtered port:

  1. On the target Linux host, flush existing rules: `sudo iptables -F`
    2. Add a rule to drop all traffic to port 8080: `sudo iptables -A INPUT -p tcp –dport 8080 -j DROP`
    3. Verify the rule: `sudo iptables -L INPUT -v`
    4. From Kali, scan port 8080: `sudo nmap -p8080 192.168.1.9`
    5. Nmap reports `filtered` after waiting for multiple probe retransmissions (no SYN/ACK or RST).
  2. To simulate a reject instead of drop (which may show as closed/filtered), use: `sudo iptables -A INPUT -p tcp –dport 8081 -j REJECT –reject-with tcp-reset`

    Explanation: Dropped packets cause Nmap to time out → filtered. Rejected packets send an RST → may appear closed. Attackers seeing `filtered` know a firewall is actively blocking the port, which itself is intelligence.

Windows equivalent (using Windows Defender Firewall):

– `New-1etFirewallRule -DisplayName “BlockPort8080” -Direction Inbound -LocalPort 8080 -Protocol TCP -Action Block` → then scan from Kali; Nmap will show `filtered` or `filtered|closed` depending on response behavior (Windows by default responds with RST for blocked ports unless configured to drop).

3. Identifying Unfiltered Ports with ACK Scans

The `unfiltered` state is unique to the TCP ACK scan (-sA). It maps firewall rules without determining if a port is open. An ACK probe receives an RST back whether the port is open or closed, but if a firewall filters it, no RST arrives.

Step‑by‑step ACK scan to detect unfiltered ports:

  1. On the target, keep the drop rule for port 8080 and allow port 80.
  2. From Kali, run: `sudo nmap -sA -p80,8080,9999 192.168.1.9`

3. Output:

  • Port 80 → `unfiltered` (RST received, firewall passes)
  • Port 8080 → `filtered` (no RST, dropped by iptables)
  • Port 9999 (closed but not filtered) → `unfiltered` (RST received)
  1. Unfiltered means the port is reachable, but Nmap cannot determine open/closed without further SYN scans.

Why use ACK scans? They map firewall rulesets (stateless vs. stateful) and identify which ports are not filtered, helping attackers focus subsequent SYN scans only on unfiltered ports.

Linux command to combine SYN and ACK:

– `sudo nmap -sS -sA -p- 192.168.1.9` (separate passes, but useful for scripting).

4. Dealing with Ambiguous States: open|filtered and closed|filtered

UDP scans often return `open|filtered` because many services do not reply to empty UDP probes, and firewalls silently drop UDP packets. Similarly, certain TCP configurations (e.g., rate limiting) produce closed|filtered.

Step‑by‑step to generate open|filtered:

  1. On target, open a UDP service (e.g., netcat): `nc -ul -p 9999`
    2. From Kali, scan UDP port: `sudo nmap -sU -p9999 192.168.1.9`
    3. Nmap may report `open|filtered` if no UDP response is received (common).
  2. Force a response using `-sU –max-retries 1` and a custom probe: sudo nmap -sU -p9999 --script=discovery 192.168.1.9.
  3. For TCP closed|filtered, apply a rate-limiting iptables rule:
    `sudo iptables -A INPUT -p tcp –dport 2222 -m limit –limit 1/s -j ACCEPT` then `-j DROP` for excess. Nmap scans may show closed|filtered.

Mitigation: Use `-sV` version detection to resolve ambiguity; if a service responds, it’s truly open. Otherwise, treat as filtered.

5. Mitigation Strategies – Controlling What Attackers Learn

Defenders can manipulate Nmap results by applying global filtering policies, rate limiting, and active response spoofing.

Step‑by‑step to harden a Linux server:

  1. Drop all incoming TCP packets by default, then whitelist only necessary ports: 
    sudo iptables -P INPUT DROP
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT  SSH
sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT  HTTPS
  2. Use `-j REJECT –reject-with icmp-port-unreachable` for UDP to avoid open|filtered ambiguity (forces a response).
  3. Implement port knocking or a stealth firewall that drops probes silently → all ports appear filtered.
  4. For Windows, enable “Stealth Mode” via advanced firewall: Block all inbound connections by default, create allow rules, and enable “Windows Firewall with Advanced Security” logging.
  5. Deploy IDS/IPS (Snort/Suricata) to detect and spoof RST packets for non-whitelisted ports, confusing Nmap into reporting `closed` even when a service exists.

Example of spoofing RST with iptables (deception):

– `sudo iptables -A INPUT -p tcp –dport 80 -j REJECT –reject-with tcp-reset` → makes an open port appear closed.

6. Advanced Evasion: Combining States for Firewall Fingerprinting

By analyzing port state patterns, pentesters can infer firewall type (stateful vs. stateless, rate‑limiting, deep packet inspection). The sequence of SYN, ACK, and FIN scans reveals how filtering rules are structured.

Step‑by‑step firewall fingerprinting:

  1. Run a SYN scan on 1000 common ports: `sudo nmap -sS -p1-1000 192.168.1.9 -oA syn_scan`
    2. Run an ACK scan on the same range: `sudo nmap -sA -p1-1000 192.168.1.9 -oA ack_scan`
    3. Compare results: if a port shows `unfiltered` in ACK but `filtered` in SYN, the firewall is stateless (allowing ACK without SYN).
  2. To detect rate limiting, use `–min-rate 5000` and observe inconsistent `filtered` or `closed|filtered` states.
  3. Use Nmap’s `-sF` (FIN scan) and `-sN` (Null scan) to bypass non‑stateful firewalls; closed ports return RST, open/filtered give no response.

Linux command to automate comparison:

nmap -sS -p1-1000 192.168.1.9 -oG syn.gnmap
nmap -sA -p1-1000 192.168.1.9 -oG ack.gnmap
grep "Unfiltered" ack.gnmap | cut -d' ' -f2 > unfiltered_ports.txt

What Undercode Say:

  • Key Takeaway 1: Nmap port states are not absolute truth—they are artifacts of packet response patterns shaped by firewalls, rate limits, and service behavior. A “filtered” port tells an attacker there is a security control in place, which itself is a valuable reconnaissance finding.
  • Key Takeaway 2: Pentesters must combine multiple scan types (SYN, ACK, UDP, FIN) to resolve ambiguous states like open|filtered. Defenders can leverage this same ambiguity by dropping probes silently or spoofing RST replies, turning Nmap’s own logic into a deceptive shield.

Analysis (approx. 10 lines): The original Hacking Articles post correctly emphasizes that port states go beyond binary open/closed, a nuance often missed by beginners. By demonstrating how iptables can produce filtered, unfiltered, and ambiguous states on demand, the content bridges theory with hands-on practice. The inclusion of Wireshark captures grounds each state in concrete packet evidence, which is essential for forensic understanding. However, the post stops short of exploring evasion techniques—such as combining ACK scans after SYN scans to detect stateful firewalls—or the defensive angle of spoofing RST replies. Expanding into these areas would elevate the material from basic reconnaissance to advanced adversary simulation. Overall, the lab-centric approach is highly effective for learning, and the step-by-step commands make it reproducible. One improvement would be adding Windows firewall equivalents, as many enterprise environments rely on Windows hosts. The mitigation strategies section is solid but could benefit from real-world examples like cloud security groups (AWS NACLs, Azure NSGs) which exhibit similar state behaviors. Despite these gaps, the post serves as an excellent reference for both red and blue teams.

Prediction:

  • +1 Defenders will increasingly adopt “stateful deception” – firewalls that dynamically alter responses (RST vs. drop vs. ICMP) based on scan patterns, turning Nmap’s state machine into a data exfiltration channel for attacker fingerprinting.
  • -1 As AI‑powered reconnaissance tools become common, attackers will use machine learning to correlate ambiguous port states across multiple scan types (SYN, ACK, FIN, UDP), reducing false positives and automatically inferring service existence even behind silent filters.
  • +1 Cloud providers (AWS, Azure, GCP) will standardize “port state telemetry” in their native security logging, allowing defenders to visualize how external scanners perceive their perimeters – a shift from reactive blocklisting to proactive recon visibility.
  • -1 The rise of encrypted TLS‑based port knocking and single‑packet authorization will make traditional Nmap scans less effective, forcing red teams to move away from port‑state analysis toward out‑of‑band signaling and covert channels.

🎯Let’s Practice For Free:

🎓 Live Courses & Certifications:

Join Undercode Academy for Verified Certifications

🚀 Request a Custom Project:

Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands

IT/Security Reporter URL:

Reported By: Nmap For – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅

🔐JOIN OUR CYBER WORLD [ CVE News • HackMonitor • UndercodeNews ]

💬 Whatsapp | 💬 Telegram

📢 Follow UndercodeTesting & Stay Tuned:

𝕏 formerly Twitter 🐦 | @ Threads | 🔗 Linkedin | 🦋BlueSky

Related Posts: