Listen to this Post
Introduction:
As artificial intelligence workloads scale from massive GPU clusters training large language models to globally distributed inference serving millions of users, the network is no longer a passive pipe—it is the critical determinant of AI performance and reliability. Cogent Communications CEO Dave Schaeffer’s keynote at NANOG 97 delivered a stark message: traditional network reliability models designed for 5 nines of uptime at Layer 1 are insufficient for AI, where network instability can throttle GPU utilization and silently degrade multi-million-dollar compute investments. This article distills Schaeffer’s vision into actionable engineering practices for network operators preparing for the tokenized era.
Learning Objectives:
- Understand the fundamental differences between network requirements for AI training versus AI inference workloads
- Learn how to redefine Service Level Objectives (SLOs) to prevent network-induced GPU throttling
- Master monitoring and automation techniques for AI-driven network environments
- Implement practical configuration strategies for low-latency, high-reliability AI infrastructure
You Should Know:
- AI Training vs. Inference: Two Networks, Two Reliability Models
Schaeffer’s core argument is that AI training and AI inference impose fundamentally different demands on network infrastructure, and treating them identically is a recipe for failure.
AI Training Workloads are characterized by massive, sustained data transfers between storage locations and GPU clusters. Since training data and compute are rarely co-located, networks must accommodate continuous, high-volume transmission with zero tolerance for interruption. The disproportionate cost of compute versus transport—where a single GPU hour can cost hundreds of dollars—means that even microsecond-scale network-induced pauses create significant financial waste. Schaeffer argues that training requires at minimum Layer 1 reliability at 5 nines (99.999%), but the economics will push organizations to improve beyond 5 nines to maximize GPU efficiency.
AI Inference Workloads, by contrast, are latency-sensitive and geographically distributed. When mature models are deployed for end-user applications, reliability challenges are amplified by the need to keep dispersed locations synchronized and available. Inference traffic often involves users uploading incremental data, shifting the directionality of network traffic from centralized to distributed models. Unlike training, where bulk throughput dominates, inference demands consistent low latency with minimal jitter—packet loss or buffering directly translates to degraded user experience.
Step-by-Step Implementation: Differentiating Your Network for AI:
- Audit existing traffic patterns to classify current and projected AI workloads as training-dominant or inference-dominant
- Define separate SLOs for each workload type—prioritize throughput and zero-loss for training, latency and jitter minimization for inference
- Implement traffic engineering using MPLS TE or Segment Routing to create dedicated paths for training versus inference traffic
- Monitor GPU utilization metrics alongside network performance to correlate network events with compute efficiency
Linux Command: Monitoring Network Latency and Jitter for Inference Traffic
Continuous latency monitoring with timestamped output for inference paths
ping -i 0.2 -D -O <inference-endpoint-ip> | while read line; do echo "$(date +%s.%N) $line"; done
Measure jitter (variation in latency) using iperf3 in UDP mode
iperf3 -c <inference-endpoint-ip> -u -b 100M -t 60 -J | jq '.intervals[].sum.jitter_ms'
Track TCP retransmissions as a proxy for network-induced GPU throttling
ss -ti | grep -E "rtt|retrans" | awk '{print $NF}'
Windows Command: Latency and Path Analysis
Continuous ping with timestamp for inference monitoring
ping -t <inference-endpoint-ip> | ForEach-Object { "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss.fff') $_" }
Path MTU discovery to identify fragmentation issues affecting throughput
ping -f -l 1472 <endpoint-ip>
Monitor TCP statistics for retransmission rates
netsh interface tcp show global
Get-1etTCPConnection | Where-Object {$_.State -eq "Established"} | Measure-Object
2. Redefining Service Level Objectives: Beyond 5 Nines
Network operators have traditionally measured reliability at Layer 1 (physical) and Layer 3 (IP routing), with 5 nines (99.999%) as the gold standard. Schaeffer argues this framework is obsolete for AI. AI applications will increasingly pressure service providers to expand their SLAs to cover higher levels of the OSI stack—Layers 4 through 7—because application-layer performance is what ultimately determines AI efficacy.
The key insight is that network instability throttles GPUs through TCP/IP’s inherent buffering mechanisms. When GPUs are fully utilized, any network-induced delay forces buffering that reduces GPU idle capacity. Defined latency becomes critical—not just average latency, but predictable, bounded latency that allows GPU scheduling to operate efficiently.
Step-by-Step Guide: Evolving Your SLO Framework for AI:
- Map AI application dependencies from Layer 1 through Layer 7 to identify where network degradation impacts application performance
- Define latency SLOs as percentiles (e.g., p99 latency < 10ms) rather than averages, with clear thresholds for GPU throttling
- Implement active measurement probes that simulate AI traffic patterns (bursty, bidirectional, mixed TCP/UDP)
- Correlate network SLO breaches with GPU utilization metrics to establish causal relationships
- Build automated remediation that triggers when SLOs approach violation thresholds
Network Device Configuration (Cisco IOS-XE): Setting Up SLA Monitoring for AI Paths
! Define an IP SLA operation measuring latency and jitter for AI inference ip sla 100 udp-jitter <inference-server-ip> 5000 source-ip <source-ip> frequency 10 history distributions-of-statistics-kept 5 history lives-kept 10 history filter all ! ! Schedule the SLA operation ip sla schedule 100 life forever start-time now ! ! Track SLO compliance with a tracking object track 100 ip sla 100 reachability delay down 3 up 5 ! ! Trigger policy-based routing or notifications on SLO breach event manager applet SLO_BREACH event track 100 state down action 1.0 syslog msg "AI Inference SLO Breached - GPU Throttling Risk" action 2.0 mail to "[email protected]" subject "AI SLO Breach"
- The Tokenized Era: Understanding AI’s Native Data Format
Schaeffer introduces the concept of “The Tokenized Era” to describe how AI fundamentally transforms data transmission. For data to be effectively used for either training or inference, it must be tokenized—converted into the discrete units that AI models process. This tokenization may occur at multiple points: with users, at edge sites of service providers, or at more centralized locations.
The engineering implication is profound: tokens will be transmitted using different protocols depending on context—sometimes traditional TCP/IP, other times at the native Layer (bypassing TCP/IP overhead entirely). Network engineers must understand where tokenization occurs in their infrastructure and design accordingly. This may involve implementing protocol optimizations, reducing encapsulation overhead, or even deploying specialized AI-optimized network hardware.
Step-by-Step Guide: Preparing Your Network for Tokenized Traffic:
- Identify tokenization points in your AI pipeline—user devices, edge POPs, or centralized data centers
- Measure token transmission efficiency for both TCP/IP and native Layer transport options
- Implement Quality of Service (QoS) policies that prioritize tokenized traffic over general internet traffic
- Consider deploying RDMA (Remote Direct Memory Access) or similar technologies for GPU-to-GPU token transfers
- Test protocol alternatives (e.g., QUIC vs. TCP) for token transmission across different network segments
Linux Configuration: Optimizing TCP for Token Transmission
Increase TCP buffer sizes for high-throughput token transfer (training) sysctl -w net.core.rmem_max=134217728 sysctl -w net.core.wmem_max=134217728 sysctl -w net.ipv4.tcp_rmem="4096 87380 134217728" sysctl -w net.ipv4.tcp_wmem="4096 65536 134217728" Enable TCP BBR congestion control for inference (low latency) sysctl -w net.core.default_qdisc=fq sysctl -w net.ipv4.tcp_congestion_control=bbr Reduce TCP acknowledgment delay for latency-sensitive inference sysctl -w net.ipv4.tcp_no_metrics_save=1 sysctl -w net.ipv4.tcp_slow_start_after_idle=0 Verify settings sysctl net.ipv4.tcp_congestion_control sysctl net.core.rmem_max
4. Data-Driven Automation: Handling AI’s Exponential Traffic Growth
As AI data volumes grow, manual network operations become impossible. Schaeffer emphasizes that a data-driven approach enables the use of automation or code to handle traffic loads. This is not optional—it is a survival requirement for network operators supporting AI workloads.
The automation strategy must address three distinct challenges: capacity planning (predicting when AI traffic will exceed available bandwidth), traffic engineering (dynamically routing AI traffic around congestion), and failure recovery (automatically rerouting AI workloads within milliseconds of a network event).
Step-by-Step Guide: Building an AI-Ready Network Automation Stack:
- Deploy telemetry collection using gNMI or NetFlow to gather real-time network state data
- Implement a time-series database (e.g., Prometheus, InfluxDB) for storing historical network performance data
- Train anomaly detection models on historical data to identify patterns preceding network degradation
- Build automation scripts that adjust routing, QoS, or capacity based on telemetry triggers
- Test automation in staging environments before production deployment, with rollback capabilities
Python Automation: Dynamic Traffic Engineering for AI Workloads
!/usr/bin/env python3
"""
Dynamic traffic engineering for AI workloads - monitors latency and adjusts routing
Requires: netmiko, requests, prometheus-api-client
"""
from netmiko import ConnectHandler
from prometheus_api_client import PrometheusConnect
import time
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(<strong>name</strong>)
Prometheus connection for telemetry
prom = PrometheusConnect(url="http://prometheus:9090", disable_ssl=True)
Device connection parameters
device = {
'device_type': 'cisco_ios',
'host': 'core-router-01',
'username': 'automation',
'password': 'secure_password'
}
def get_path_latency(source, destination):
"""Query Prometheus for current path latency"""
query = f'latency_seconds{{source="{source}",dest="{destination}"}}'
result = prom.custom_query(query)
if result:
return float(result[bash]['value'][bash])
return None
def adjust_qos_for_ai_traffic(latency_ms):
"""Adjust QoS policies based on current latency"""
connection = ConnectHandler(device)
if latency_ms > 50:
High latency - prioritize AI traffic
config_commands = [
'class-map match-any AI-TRAFFIC',
'match protocol http',
'match protocol https',
'match dscp 46',
'policy-map AI-POLICY',
'class AI-TRAFFIC',
'bandwidth percent 50',
'priority level 1'
]
else:
Normal latency - standard QoS
config_commands = [
'policy-map AI-POLICY',
'class AI-TRAFFIC',
'bandwidth percent 30'
]
output = connection.send_config_set(config_commands)
connection.disconnect()
logger.info(f"QoS adjusted: {output}")
return output
Main automation loop
if <strong>name</strong> == "<strong>main</strong>":
while True:
latency = get_path_latency("dc-east", "dc-west")
if latency:
logger.info(f"Current latency: {latency1000:.2f}ms")
if latency > 0.05: 50ms threshold
adjust_qos_for_ai_traffic(latency 1000)
time.sleep(60)
- Monitoring for Peak Performance: The AI-Specific Observability Stack
Traditional network monitoring—ping checks, interface utilization graphs, and syslog aggregation—is insufficient for AI workloads. Schaeffer stresses that monitoring tools must ensure peak performance and minimize outages. AI-specific observability requires GPU-level visibility, application-layer performance metrics, and predictive analytics.
The critical metric is GPU utilization as a function of network performance. If network latency increases by 5ms, how much GPU capacity is lost? If packet loss reaches 0.1%, what is the impact on training convergence time? Answering these questions requires integrating network monitoring with compute infrastructure monitoring.
Step-by-Step Guide: Building an AI-Observability Pipeline:
- Deploy network performance monitors (e.g., Kentik, ThousandEyes) that measure latency, jitter, and loss along AI traffic paths
- Integrate with GPU monitoring tools (e.g., NVIDIA DCGM, Prometheus GPU exporter) to correlate network and compute metrics
- Create dashboards that show network performance alongside GPU utilization, training throughput, and inference latency
- Set up alerting based on composite conditions (e.g., network latency > threshold AND GPU utilization dropping)
- Implement distributed tracing to follow AI requests from user through network to inference endpoint
Prometheus Configuration: GPU and Network Monitoring Integration
prometheus.yml - Scrape configuration for AI monitoring global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: Network device telemetry via gNMI - job_name: 'network-telemetry' static_configs: - targets: ['core-router-01:50051', 'core-router-02:50051'] metrics_path: /gnmi params: target: ['10.0.0.1'] GPU metrics from NVIDIA DCGM - job_name: 'gpu-metrics' static_configs: - targets: ['gpu-server-01:9400', 'gpu-server-02:9400'] AI application performance - job_name: 'ai-application' static_configs: - targets: ['inference-endpoint:8080'] metrics_path: /actuator/prometheus Alerting rules for AI performance degradation rule_files: - "ai_alerts.yml"
Alerting Rules (ai_alerts.yml):
groups:
- name: ai_performance
rules:
- alert: GPUIThrottlingDetected
expr: (gpu_utilization < 0.7) and (network_latency_seconds > 0.01)
for: 5m
annotations:
summary: "GPU throttling detected - network latency causing idle GPUs"
description: "Network latency {{ $value }}s is causing GPU utilization to drop below 70%"
<ul>
<li>alert: InferenceLatencySpike
expr: histogram_quantile(0.99, inference_latency_seconds_bucket) > 0.1
for: 2m
annotations:
summary: "Inference p99 latency exceeds 100ms"
description: "AI inference latency is degrading user experience"
- Geographically Distributed Inference: Keeping the World in Sync
When mature AI models are deployed for production use, reliability challenges are amplified by the need to keep geographically dispersed locations synchronized and consistently available. This is the distributed systems problem applied to AI: how do you maintain model consistency, data freshness, and low-latency response across a global network?
Schaeffer’s framework suggests that distributed inference requires fundamentally different network architecture than centralized training. Edge locations must cache models, synchronize updates, and handle user uploads—all while maintaining sub-100ms latency to end users.
Step-by-Step Guide: Designing for Distributed AI Inference:
- Map user locations and identify optimal edge deployment points for inference
- Design a model distribution strategy—centralized training with periodic model updates to edge locations
- Implement anycast routing for inference endpoints to direct users to the nearest available instance
- Deploy global load balancing that considers both geography and current network conditions
- Establish data synchronization protocols between edge locations and central data stores
Anycast Configuration for AI Inference (BGP):
! Router configuration for anycast inference endpoint interface Loopback100 description AI-Inference-Anycast ip address 203.0.113.100 255.255.255.255 ! router bgp 65001 network 203.0.113.100 mask 255.255.255.255 neighbor 192.0.2.1 route-map ADVERTISE-ANYCAST out ! route-map ADVERTISE-ANYCAST permit 10 set community 65001:100 set local-preference 150
Synchronization Monitoring Script:
!/bin/bash
Monitor model version consistency across inference endpoints
ENDPOINTS=("inference-1yc" "inference-lon" "inference-tokyo" "inference-syd")
EXPECTED_VERSION=$(curl -s https://model-registry.internal/current-version)
for endpoint in "${ENDPOINTS[@]}"; do
VERSION=$(curl -s "https://${endpoint}.internal/version" | jq -r '.model_version')
if [ "$VERSION" != "$EXPECTED_VERSION" ]; then
echo "ALERT: ${endpoint} running model version ${VERSION}, expected ${EXPECTED_VERSION}"
Trigger automated model update
curl -X POST "https://${endpoint}.internal/update" -d "version=${EXPECTED_VERSION}"
else
echo "OK: ${endpoint} synchronized"
fi
done
What Undercode Say:
- The network is the new bottleneck. No matter how many GPUs you throw at AI, network-induced latency and packet loss will throttle performance. The economics are clear: transport is cheap compared to compute, so over-investing in network reliability pays dividends in GPU efficiency.
-
Traditional SLOs are obsolete. Five nines at Layer 1 and three nines at Layer 3 are insufficient for AI. Operators must expand SLAs to cover Layers 4-7 and define latency percentiles that prevent GPU throttling.
-
Tokenization changes everything. AI’s native data format—tokens—will be transmitted using different protocols in different contexts. Network engineers must understand where tokenization occurs and optimize accordingly.
-
Automation is non-1egotiable. AI data volumes grow exponentially; manual operations cannot scale. Data-driven automation with telemetry, machine learning, and programmatic traffic engineering is the only viable path forward.
-
Training and inference are two different networks. Treating them identically is a strategic error. Training demands bulk throughput and zero loss; inference demands low, predictable latency and global distribution.
Prediction:
+1: Network operators who adopt AI-specific SLOs and monitoring will capture significant market share as enterprises seek reliable AI infrastructure partners.
+1: The emergence of “AI-optimized networks” will create a new category of network services, with premium pricing for guaranteed latency and zero-loss training paths.
+1: Automation and telemetry will become mandatory skills for network engineers, with AI operations displacing traditional CLI-based management within 3-5 years.
-1: Organizations that fail to evolve their network reliability models for AI will waste millions in underutilized GPU capacity and lose competitive advantage.
-1: The complexity of managing distributed inference will create new attack surfaces, with tokenized data transmission introducing novel security vulnerabilities.
-1: The shift toward application-layer SLAs will expose network operators to liability for AI performance degradation, fundamentally changing commercial relationships.
▶️ Related Video (86% Match):
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
IT/Security Reporter URL:
Reported By: On June – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
