TCP 3-Way Handshake: The Silent Guardian Of Every Internet Connection – And How Hackers Exploit It + Video

Introduction:

The TCP 3-way handshake is the foundation of reliable communication over the internet—but it’s also a prime target for cyberattacks like SYN floods and session hijacking. Understanding this handshake isn’t just about networking basics; it’s a critical skill for detecting anomalies, hardening firewalls, and analyzing malicious traffic in real time.

Learning Objectives:

– Analyze the SYN, SYN-ACK, ACK sequence using packet capture tools like Wireshark and tcpdump.
– Identify and mitigate TCP-based attacks, including SYN flooding and Man-in-the-Middle (MitM) handshake interception.
– Implement firewall rules and system configurations (Linux/Windows) to monitor and secure TCP connection establishments.

You Should Know:

1. Anatomy of the Handshake – From SYN to ACK

The TCP 3-way handshake establishes a connection-oriented session between a client and server. Step-by-step:
– SYN (Synchronize): Client sends a TCP segment with the SYN flag set, along with an initial sequence number (ISN), e.g., `seq=x`.
– SYN-ACK: Server responds with SYN and ACK flags, acknowledging the client’s ISN (`ack=x+1`) and sending its own ISN (`seq=y`).
– ACK: Client sends an ACK (`ack=y+1`), and the connection enters ESTABLISHED state.

How to observe it:

– Linux (tcpdump): `sudo tcpdump -i eth0 ‘tcp

 & (tcp-syn) != 0' -c 10`
- Windows (PacketMonitor): `pktmon filter add -p TCP` then `pktmon start --capture --pkt-size 0 --file-size 100`
- Wireshark filter: `tcp.flags.syn == 1 and tcp.flags.ack == 0` for initial SYN; `tcp.flags.syn == 1 and tcp.flags.ack == 1` for SYN-ACK.

<h2 style="color: yellow;">2. Detecting Handshake Anomalies with Wireshark</h2>

Suspicious patterns—like repeated SYNs without ACKs or half-open connections—indicate scanning or flood attacks.

<h2 style="color: yellow;">Step-by-step analysis:</h2>
1. Capture live traffic: `sudo tcpdump -i eth0 -w handshake.pcap`
2. Open in Wireshark, apply filter `tcp.flags.syn == 1`.
3. Use Statistics → Flow Graph to visualize handshake sequences.
<h2 style="color: yellow;">4. Look for:</h2>
- Many SYNs from one IP without completing ACK → SYN flood candidate.
- Retransmitted SYNs with different sequence numbers → OS fingerprinting.
5. Export conversation list: Statistics → Conversations → TCP → copy IPs for blocking.

Linux command to check half-open connections: `netstat -ant | grep SYN_RECV | wc -l` 
<h2 style="color: yellow;">Windows PowerShell: `Get-1etTCPConnection | Where-Object {$_.State -eq "SynReceived"}`</h2>

<h2 style="color: yellow;">3. SYN Flood Attack – Exploitation and Mitigation</h2>

Attackers send rapid SYN requests without final ACKs, exhausting the server’s backlog queue (the “listen” state). Mitigation requires tuning kernel parameters and enabling SYN cookies.

<h2 style="color: yellow;">Linux mitigation:</h2>
[bash]
 View current backlog
sysctl net.ipv4.tcp_max_syn_backlog
 Increase backlog size
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=4096
 Enable SYN cookies (no backlog overflow)
sudo sysctl -w net.ipv4.tcp_syncookies=1
 Reduce SYN-ACK retries
sudo sysctl -w net.ipv4.tcp_synack_retries=2

Windows mitigation (Registry):

– `HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters`

`SynAttackProtect` = 1 (enables SYN attack protection)

`TcpMaxHalfOpen` = 500 (default 500)

`TcpMaxHalfOpenRetried` = 400

Apply with `netsh int tcp set global security=enable` and reboot.

4. Firewall Rules to Filter Malicious Handshakes

Block incomplete handshakes or restrict SYN rates using iptables (Linux) or Windows Advanced Firewall.

Linux iptables rate limiting:

 Limit SYN packets to 10 per second from a single source
sudo iptables -A INPUT -p tcp --syn -m limit --limit 10/s -j ACCEPT
sudo iptables -A INPUT -p tcp --syn -j DROP
 Log excess SYNs
sudo iptables -A INPUT -p tcp --syn -m limit --limit 5/m -j LOG --log-prefix "SYN flood: "

Windows PowerShell (New-1etFirewallRule):

New-1etFirewallRule -DisplayName "Block High SYN Rate" -Direction Inbound -Protocol TCP -Action Block -DynamicTarget ANY -EdgeTraversalPolicy Block

(Third-party tools like `pshNetFirewall` or IDS/IPS integration required for dynamic rate limits; consider Azure Firewall or WAF.)

5. TCP Sequence Number Prediction – Session Hijacking

If sequence numbers are predictable, attackers can inject forged ACKs and take over a connection (MitM). Modern OSes use random ISNs, but legacy systems are vulnerable.

Check your OS randomness:

– Linux: `sysctl net.ipv4.tcp_timestamps` (disable for less info leak; enable randomization with `net.ipv4.tcp_challenge_ack_limit`)
– Windows: By default uses secure ISN generation since Vista/Server 2008.

Test with `hping3` (Linux):

sudo hping3 -S -p 80 --rand-source -c 5 target.com
 Observe sequence increments; predictable if constant offset

Mitigation: Use TCP-AO (TCP Authentication Option) or IPsec. Disable TCP timestamps to reduce information leakage: `sudo sysctl -w net.ipv4.tcp_timestamps=0`

6. Handshake Timeout Tuning for Performance & Security

Aggressive timeout values reduce exposure to half-open attacks but may drop legitimate slow clients.

Recommended settings:

| Parameter | Linux (sysctl) | Windows (reg/set) |

|–|-|-|

Apply with: `sudo sysctl -p` / `netsh int tcp set global timestamps=enabled`.

7. Handshake Emulation for Network Reconnaissance

Red teams and attackers use custom scripts to craft handshake packets and probe firewall rules.

Scapy (Python) example:

from scapy.all import 
syn = IP(dst="192.168.1.1")/TCP(dport=80, flags="S")
syn_ack = sr1(syn, timeout=2)
if syn_ack and syn_ack[bash].flags == 0x12:  SYN-ACK
ack = IP(dst="192.168.1.1")/TCP(dport=80, flags="A", seq=syn_ack.ack, ack=syn_ack.seq+1)
send(ack)
print("Handshake completed")

Detection: Monitor for incomplete handshakes from single IPs. Use `snort` rule: `alert tcp any any -> $HOME_NET any (flags:S; msg:”SYN scan”; threshold:type both, track by_src, count 5, seconds 10);`

What Undercode Say:

– The TCP 3-way handshake is both a reliability feature and an attack surface—engineers must treat it as a security control, not just a protocol detail.
– Most modern breaches start with reconnaissance using malformed or half-open handshakes; proactive monitoring with tcpdump/Wireshark and tuned SYN cookies reduces flood risk by 90%.
– Windows and Linux differ in default timeouts and ISN generation—cross-platform knowledge is essential for blue teams.
– Red teams can leverage handshake emulation to bypass stateless firewalls; defenders must implement stateful inspection and rate-based alerts.
– Automation: Integrate netstat/SYN_RECV alerts into SIEM (e.g., Splunk query `index=network sourcetype=tcp “SYN_RECV” | stats count by src_ip`).
– Future networks (QUIC, MPTCP) still rely on handshake principles but add encryption; legacy TCP will remain dominant in internal infrastructure for years.
– A simple SYN flood can cripple unhardened servers—always validate `net.ipv4.tcp_syncookies=1` in container and cloud environments.
– Wireshark’s “Expert Info” warnings (e.g., “TCP Spurious Retransmission”) often trace back to asymmetric routing interfering with handshake ACKs.
– For IoT devices, disabling TCP timestamps and reducing keepalive intervals prevents battery drain from half-open attacks.
– The single most overlooked command: `ss -t state syn-recv` – reveals ongoing attacks faster than any GUI tool.

Expected Output:

Introduction: The TCP 3-way handshake establishes trust before data transfer, but this “trust” is blind to malicious intent. Security professionals must analyze and harden every SYN, SYN-ACK, and ACK to prevent session hijacking and denial-of-service.

Prediction:

– +1 Adoption of eBPF-based handshake monitoring in Linux kernels (6.x+) will enable real-time SYN flood mitigation with near-zero overhead.
– -1 As TLS 1.3 and Encrypted Client Hello (ECH) expand, encrypted handshakes will make it harder to detect malicious TCP patterns, forcing defenders to rely on metadata (SYN rates, RTT variance).
– +1 Cloud-1ative firewalls (AWS Network Firewall, Azure Firewall) now include ML-based anomaly detection for TCP handshake states, reducing false positives in auto-scaling environments.
– -1 Legacy SCADA and medical devices using fixed TCP sequence numbers will remain vulnerable to session injection until air-gapped or phased out.
– +1 QUIC’s 0-RTT handshake reduces latency but introduces replay risks; TCP’s 3-way model will stay critical for teaching fundamental state machine security.

▶️ Related Video (76% Match):

🎯Let’s Practice For Free:

🎓 Live Courses & Certifications:

[Join Undercode Academy for Verified Certifications](https://undercode.co.uk/certifications/)

🚀 Request a Custom Project:

Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[[email protected]](mailto:[email protected])
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands

IT/Security Reporter URL:

Reported By: [Networking Tcpip](https://www.linkedin.com/posts/networking-tcpip-cybersecurity-share-7469742286701031424-VQDF/) – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅

🔐JOIN OUR CYBER WORLD [ CVE News • HackMonitor • UndercodeNews ]

[💬 Whatsapp](https://undercode.help/whatsapp) | [💬 Telegram](https://t.me/UndercodeCommunity)

📢 Follow UndercodeTesting & Stay Tuned:

[𝕏 formerly Twitter 🐦](https://x.com/undercodeupdate) | [@ Threads](https://www.threads.net/@undercodetesting) | [🔗 Linkedin](https://www.linkedin.com/company/undercodetesting/) | [🦋BlueSky](https://bsky.app/profile/undercode.bsky.social)

Listen to this Post