TCP 3-Way Handshake: The Secret Handshake Every Hacker Must Master (And How to Exploit It) + Video

By /

Listen to this Post

Featured Image

Introduction:

Before any reliable TCP communication can occur, a client and server must establish a connection through a process known as the TCP 3-Way Handshake. This exchange of SYN, SYN-ACK, and ACK packets forms the bedrock of network reliability, but it also introduces attack surfaces that every cybersecurity professional—from SOC analysts to red teamers—must understand to defend or ethically exploit network traffic.

Learning Objectives:

  • Understand the three-step TCP handshake sequence and its role in establishing reliable connections.
  • Capture and analyze handshake packets using command-line tools (tcpdump, ss, netstat) and Wireshark.
  • Identify and simulate SYN flood attacks, then apply mitigation techniques such as SYN cookies and rate limiting.

You Should Know:

  1. Capturing the Live TCP Handshake with tcpdump (Linux)
    The most direct way to see the three‑way handshake is to capture packets between two endpoints. Use `tcpdump` on a Linux machine to listen for TCP traffic and isolate the handshake.

Step‑by‑step guide:

  • Open a terminal and identify your network interface: `ip link show`
    – Start a capture on that interface (e.g., eth0), filtering for a specific destination port (e.g., 80) and displaying verbosely:

    sudo tcpdump -i eth0 -1n -S port 80
  • From another terminal, initiate a TCP connection using `curl` or telnet: 
    curl http://example.com
  • Observe the three lines: `Flags 
    ` (SYN from client), `Flags [S.]` (SYN‑ACK from server), `Flags [.]` (ACK from client).</li>
</ul>

What this does: It captures raw packet headers, showing sequence numbers, flags, and window sizes. Use `-S` to see absolute sequence numbers instead of relative ones. Save captures for later analysis: <code>-w handshake.pcap</code>.

<ol>
<li>Following the Handshake in Wireshark (GUI + CLI Analysis)
Wireshark provides a visual and filter‑based approach to inspect the handshake, essential for SOC investigations and forensic analysis.</li>
</ol>

<h2 style="color: yellow;">Step‑by‑step guide:</h2>

<ul>
<li>Start a capture on the relevant interface, then filter with <code>tcp.flags.syn == 1</code>.</li>
<li>Right‑click any SYN packet and select Follow > TCP Stream. Wireshark will show the entire conversation, including the handshake at the top.</li>
<li>For command‑line analysis, use `tshark` (Wireshark’s CLI):
[bash]
tshark -r capture.pcap -Y "tcp.flags.syn==1 or tcp.flags.ack==1" -T fields -e frame.number -e ip.src -e tcp.flags
  • Identify retransmissions or missing ACKs – common signs of network issues or attack patterns.

You should know: In Windows, use `”C:\Program Files\Wireshark\tshark.exe”` similarly. The filter `tcp.analysis.flags` will highlight handshake anomalies like duplicate ACKs.

  1. Simulating a SYN Flood Attack (Ethical Testing with hping3)
    A SYN flood exploits the handshake by sending many SYN requests without completing the ACK, exhausting server resources. Use `hping3` in a lab environment to test your detection and mitigation.

Step‑by‑step guide (Linux only, isolated lab):

  • Install hping3: `sudo apt install hping3`
    – Launch a SYN flood against a test target (e.g., 192.168.1.10 port 80):

    sudo hping3 -S -p 80 --flood --rand-source 192.168.1.10
  • On the target server, monitor the half‑open connection queue: 
    netstat -an | grep :80 | grep SYN_RECV | wc -l
  • Observe how the queue fills up. Legitimate clients then experience connection timeouts.

Mitigation: Enable SYN cookies temporarily:

echo 1 > /proc/sys/net/ipv4/tcp_syncookies

SYN cookies eliminate the need to store half‑open connections, turning resource exhaustion into a computational challenge.

  1. Detecting Handshake-Based Attacks with `ss` and `netstat` (Linux & Windows)
    Both Linux and Windows provide built‑in tools to monitor TCP states and spot SYN floods or failed handshakes.

Step‑by‑step guide:

  • Linux – `ss` (modern replacement for netstat): 
    ss -tan state syn-recv

    This lists all half‑open connections. Monitor the count over time: `watch -1 1 ‘ss -tan state syn-recv | wc -l’`
    – Linux – netstat:

    netstat -s | grep -i "connection resets"
netstat -s | grep -i "syncookies"
  • Windows – PowerShell: Show TCP connections with `SYN_RECEIVED` state: 
    Get-1etTCPConnection -State SynReceived
  • Windows – classic command prompt: 
    netstat -an | find "SYN_RECEIVED"

What this does: It reveals the current backlog of incomplete handshakes. A sudden spike in SYN_RECV without corresponding ESTABLISHED connections indicates a possible SYN flood.

  1. Hardening Cloud & On‑Prem Servers Against Handshake Exploits
    Modern cloud environments (AWS, Azure, GCP) and Linux kernels offer tunables to reduce handshake vulnerabilities.

Step‑by‑step guide:

  • SYN cookies (persistent): Add to /etc/sysctl.conf: 
    net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_synack_retries = 2

Apply: `sudo sysctl -p`

  • Increase backlog queue size: 
    net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 8192
  • Rate limiting with iptables (Linux): Limit SYN packets per second from a single source: 
    iptables -A INPUT -p tcp --syn -m limit --limit 1/s -j ACCEPT
iptables -A INPUT -p tcp --syn -j DROP
  • AWS Security Groups / Azure NSG: Use “Connection tracking” to enforce TCP handshake completion before forwarding traffic.

You should know: These settings trade off between performance and security. Overly aggressive SYN limits can block legitimate burst traffic (e.g., web server flash sales).

  1. Crafting Custom Handshake Packets with Scapy (Python for Red Teaming)
    Scapy allows you to build arbitrary TCP packets, enabling deep understanding of handshake mechanics and custom exploit development.

Step‑by‑step guide:

  • Install Scapy: `pip install scapy`
    – Write a Python script to send a SYN, wait for SYN‑ACK, then reply with ACK:

    from scapy.all import </li>
</ul>

target_ip = "192.168.1.10"
target_port = 80
sport = random.randint(1024, 65535)

Send SYN
syn = IP(dst=target_ip)/TCP(sport=sport, dport=target_port, flags="S", seq=1000)
syn_ack = sr1(syn, timeout=2, verbose=0)

Send ACK
ack = IP(dst=target_ip)/TCP(sport=sport, dport=target_port, flags="A", seq=syn_ack.ack, ack=syn_ack.seq+1)
send(ack, verbose=0)
print("Handshake completed manually.")

    – Verify the connection using ss -tan | grep target_ip.

    What this does: It mimics a normal TCP client but gives you control over sequence numbers, options, and timing. Use this to test firewalls that inspect handshake validity or to bypass spoofing protections.

    7. Windows Registry Tuning for Handshake Protection

    Windows servers also expose TCP parameters to mitigate handshake attacks (SYN attack protection).

    Step‑by‑step guide (run as Administrator in PowerShell):

    • Enable SYN attack protection: 
      Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters" -1ame "SynAttackProtect" -Value 2 -Type DWord
    • Set maximum half‑open connections: 
      Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters" -1ame "TcpMaxHalfOpen" -Value 500 -Type DWord
Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters" -1ame "TcpMaxHalfOpenRetried" -Value 400 -Type DWord
    • Enable TCP timestamps (helps with RTT calculations): 
      Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters" -1ame "Tcp1323Opts" -Value 3 -Type DWord
    • Reboot or restart the TCP/IP stack: `Restart-Service NetTcpPortSharing`

      You should know: Misconfiguring these parameters can break legitimate high‑throughput connections. Always test in a staging environment.

    What Undercode Say:

    • Key Takeaway 1: The TCP 3‑Way Handshake is not just a network theory concept—it is an active attack surface where SYN floods, reflection attacks, and resource exhaustion occur.
    • Key Takeaway 2: Defenders must combine real‑time monitoring (tcpdump, ss, netstat), kernel hardening (SYN cookies, backlog tuning), and cloud‑native rate limiting to protect modern infrastructures.

    Analysis: The post from Ethical Hackers Academy correctly emphasizes that mastering the handshake is foundational for cybersecurity. However, most beginners stop at memorizing SYN→SYN‑ACK→ACK without understanding how to capture, simulate, or break the handshake. This article bridges that gap by providing executable commands for both Linux and Windows, plus attack simulation with hping3 and packet crafting with Scapy. For SOC analysts, the ability to filter `tcp.flags.syn==1` in Wireshark and correlate with `ss -tan state syn-recv` transforms abstract theory into actionable detection. Meanwhile, red teamers can leverage the same knowledge to craft stealthy scans that never complete the handshake, evading basic connection‑based logging. The inclusion of cloud hardening (SYN cookies, iptables rate limits) and Windows registry tuning makes this relevant across hybrid environments. Ultimately, the handshake is a double‑edged sword: reliable communication depends on it, but so do some of the simplest DDoS attacks.

    Prediction:

    • -1 As IoT and edge devices proliferate, poorly tuned TCP stacks will remain vulnerable to SYN floods, allowing botnets to amplify resource exhaustion with minimal bandwidth.
    • +1 AI‑powered network anomaly detection (e.g., using recurrent neural networks on packet time series) will automatically classify abnormal handshake patterns, reducing false positives and enabling real‑time dynamic SYN cookie activation.
    • -1 Cloud native environments (Kubernetes, service meshes) often abstract away TCP tuning, leading to silent handshake failures that are misdiagnosed as application bugs rather than network‑level exhaustion.
    • +1 The adoption of TCP Fast Open (TFO) and QUIC (which reduces handshake rounds) will gradually replace the classic 3‑way handshake for modern web traffic, shrinking the attack surface while introducing new cryptographic handshake vulnerabilities.

    ▶️ Related Video (78% Match):

    🎯Let’s Practice For Free:

    🎓 Live Courses & Certifications:

    Join Undercode Academy for Verified Certifications

    🚀 Request a Custom Project:

    Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
    [email protected]
    💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands

    IT/Security Reporter URL:

    Reported By: Tcpip Networking – Hackers Feeds
    Extra Hub: Undercode MoN
    Basic Verification: Pass ✅

    🔐JOIN OUR CYBER WORLD [ CVE News • HackMonitor • UndercodeNews ]

    💬 Whatsapp | 💬 Telegram

    📢 Follow UndercodeTesting & Stay Tuned:

    𝕏 formerly Twitter 🐦 | @ Threads | 🔗 Linkedin | 🦋BlueSky

Related Posts: