Listen to this Post
Introduction:
Port scanning is the first punch in any cyber engagement, but most analysts only see Nmap’s text output—not the raw packets flying across the wire. This article pairs five essential Nmap scan types (TCP Connect, SYN stealth, FIN, NULL, and Xmas) with live Wireshark captures, revealing exactly how open vs. closed ports respond and how defenders can detect each technique.
Learning Objectives:
- Differentiate between full‑handshake (‑sT) and half‑open (‑sS) scans by analyzing TCP flags and RST behavior.
- Detect FIN, NULL, and Xmas scans using packet‑level anomalies and signature‑based rules.
- Apply Wireshark display filters and command‑line tools to identify malicious reconnaissance in real traffic.
You Should Know:
- TCP Connect Scan (–sT) – The Loud but Reliable Handshake
The TCP Connect scan uses the operating system’s `connect()` system call to complete the full three‑way handshake. It works without root privileges but leaves a full connection record in target logs.
Step‑by‑step analysis against an open port (22/SSH):
Run a TCP Connect scan on port 22 sudo nmap -sT -p 22 192.168.1.5
Wireshark filter: `ip.addr == 192.168.1.5 and tcp.port in {22 21}`
What happens on the wire (open port):
1. Attacker (192.168.1.17) → Target: SYN (Seq=0)
2. Target → Attacker: SYN/ACK (Seq=0, Ack=1)
3. Attacker → Target: ACK (Ack=1)
- Attacker → Target: RST/ACK – Nmap immediately tears down the connection
Closed port (21/FTP): Only two packets – SYN from attacker, RST/ACK from target. The RST flag indicates no listener.
Detection using tcpdump:
sudo tcpdump -i eth0 'tcp[bash] & (tcp-syn) != 0 and tcp[bash] & (tcp-ack) == 0'
- SYN Stealth Scan (–sS) – The Default Root Scan
SYN scan crafts raw packets and never completes the handshake, avoiding application‑layer logging. Requires raw socket privileges.
Command and Wireshark observation:
sudo nmap -sS -p 22 192.168.1.5
Open port behavior:
Attacker sends SYN → Target replies SYN/ACK → Attacker immediately sends RST (never ACK). No three‑way handshake completion.
Closed port behavior:
SYN → RST/ACK (same as Connect scan).
Detect SYN scans with iptables:
Log SYN packets that are followed by RST from your host iptables -A INPUT -p tcp --tcp-flags SYN,ACK SYN -m state --state NEW -j LOG --log-prefix "SYN_SCAN "
3. FIN Scan (–sF) – Bypassing Stateless Firewalls
FIN scans violate RFC 793 by sending only the FIN flag without an established connection. Many legacy firewalls and non‑Windows hosts respond differently.
Command:
sudo nmap -sF -p 22,21 192.168.1.5
Open port (Linux target): No response – the packet is silently dropped (because no matching socket). Nmap marks as open|filtered.
Closed port: Target replies with RST/ACK – definitive “closed” signal.
Wireshark filter: `tcp.flags.fin == 1 and tcp.flags.syn == 0 and tcp.flags.ack == 0`
Windows exception: Windows systems (up to Server 2016) ignore all FIN probes, making them appear fully filtered.
4. NULL Scan (–sN) – All Flags Off
NULL scan sets no TCP flags (zero flags). It exploits the same RFC gap as FIN scans.
Execution:
sudo nmap -sN -p 80,443 192.168.1.5
Response logic (Linux/Unix):
- Open port → no reply (filtered/open)
- Closed port → RST/ACK
Custom detection with tcpdump:
sudo tcpdump -i eth0 'tcp[bash] == 0' NULL flag pattern
5. Xmas Scan (–sX) – All Flags Lit
Xmas scan sets FIN, PSH, and URG flags simultaneously – the “Christmas tree” packet.
Command:
sudo nmap -sX -p 22 192.168.1.5
Behavior mirrors FIN and NULL scans on Linux: open ports stay silent, closed ports return RST.
Snort rule to detect all three stealth scans:
alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"SCAN FIN/NULL/XMAS"; flow:stateless; flags: F,12; threshold: type both, track by_src, count 5, seconds 10; sid:1000001;)
6. Defensive Hardening Against Nmap Scans
While you cannot stop a determined scanner, you can reduce noise and log everything.
Linux: Use `iptables` to limit SYNs per second:
iptables -A INPUT -p tcp --dport 22 -m limit --limit 1/s -j ACCEPT iptables -A INPUT -p tcp --dport 22 -j DROP
Windows (PowerShell – built‑in firewall):
New-1etFirewallRule -DisplayName "Block Stealth Scans" -Direction Inbound -Protocol TCP -Action Block -RemoteAddress Any
Enable logging of dropped packets (Linux):
iptables -A INPUT -m state --state INVALID -j LOG --log-prefix "INVALID_STATE "
7. Hands‑On Lab: Recreate the Analysis
Set up two VMs (Kali attacker + Linux target). Run each scan with `nmap` while capturing with Wireshark.
Wireshark coloring rule for scan flags:
Go to View → Coloring Rules → Add. Name: “SYN only”, Filter: tcp.flags.syn==1 && tcp.flags.ack==0 && tcp.flags.fin==0, Foreground: Red.
Compare output using `tshark` (command‑line Wireshark):
tshark -r scan.pcap -Y "tcp.flags.syn==1 and tcp.flags.ack==0" -T fields -e ip.src -e tcp.srcport -e tcp.dstport
What Undercode Say:
- Key Takeaway 1: Open vs. closed port responses are consistent across all five scan types – closed ports always reply with RST/ACK, while open ports either complete handshake (‑sT) or drop packets (‑sS, ‑sF, ‑sN, ‑sX). This deterministic behavior allows Nmap to infer states even without a full ACK.
- Key Takeaway 2: FIN, NULL, and Xmas scans are relics of the 1990s firewall evasion – they work only against older or misconfigured Unix‑like systems. Modern stateful firewalls and Windows 10+ drop these packets without distinction, making them less reliable but still useful for fingerprinting.
- Analysis: The Wireshark comparison table in the original document (pages 13‑14) is a goldmine for SOC analysts: it maps each scan type to its exact packet exchange. Understanding these micro‑patterns turns raw packet dumps into actionable intelligence. For red teamers, the SYN stealth scan remains the champion of speed and stealth – but only if you control your RST timing. For blue teams, monitoring high rates of SYN‑only packets or unusual flag combinations (FIN without ACK) provides near‑zero‑false‑positive alerts for active scanning.
Prediction:
- -P Automated red teaming tools will integrate real‑time Wireshark feedback loops, adjusting scan throttle and flag patterns based on target’s RST behavior – turning static nmap commands into adaptive reconnaissance engines.
- +1 Cloud providers (AWS, Azure) are rolling out AI‑driven traffic anomaly detectors that learn baseline flag distributions; FIN/NULL/Xmas scans will trigger instant flagging within 3‑5 packets, making traditional stealth scans obsolete by 2027.
- -1 As IoT and OT devices continue using legacy TCP stacks (many still RFC 793‑only), FIN/NULL scans will remain effective against industrial controllers and medical devices, creating an enduring blind spot for defenders who rely on modern firewalls alone.
▶️ Related Video (76% Match):
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
IT/Security Reporter URL:
Reported By: Ethical Hacking – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
