Listen to this Post
Introduction:
Enterprises no longer have a single AI front door – they have dozens. Claude Code, Codex, Cursor, ChatGPT, Gemini, and internal agents each connect directly to model providers with separate configurations, separate spend, and zero shared visibility. This fragmentation, known as AI sprawl, compounds rapidly: every new tool brings another direct line to a model provider, another set of permissions to configure, and another gap in visibility. Glean AI Gateway addresses this by sitting as a single layer underneath all AI tools, centralizing model access, provider routing, spend quotas, and audit trails across every surface.
Learning Objectives:
- Understand the architectural components of an enterprise AI gateway and how they solve AI sprawl
- Learn to implement centralized LLM access control, spend governance, and observability
- Master MCP (Model Context Protocol) Gateway configuration for standardized tool access across AI systems
You Should Know:
- The AI Sprawl Problem – Why Your Current Setup Is Broken
Every developer on your team is using different AI tools. Each tool talks directly to its own model provider. No one knows which team is spending what, on which model, through which app. Security policies that apply to one tool don’t automatically apply to another. When a new tool gets added, the entire setup starts over.
This isn’t just an inconvenience – it’s a governance nightmare. Without a centralized layer, there’s no reliable way to prevent expensive frontier models from being used for lightweight tasks, or to stop token spend from running over budget before anyone notices. Gartner predicts that 40% of enterprise applications will embed task-specific AI agents by the end of 2026, up from less than 5% in 2025. The sprawl is only going to accelerate.
The solution requires two complementary components: an LLM Gateway that centralizes model access, provider routing, spend quotas, and audit trails, and an MCP Gateway that standardizes which tools and actions AI systems can access.
- LLM Gateway Architecture – The Control Plane for Model Access
The LLM Gateway serves as a standardized abstraction layer between your applications and multiple AI model providers. It functions similarly to an API gateway, providing a unified access point that standardizes interactions across different provider interfaces while enabling centralized management and monitoring.
Key Capabilities:
- Model Agnostic Support: Supports 30+ models across open source and proprietary providers including Claude, Gemini, OpenAI, NVIDIA, and more
- Intelligent Routing: Route requests to the right model based on task complexity, cost, and performance requirements
- Spend Controls: Set budget limits and quotas per user, per app, with automatic fallback to less expensive models when limits are approached
- Unified Observability: Token spend broken down by model and app, model latency and error rates in one operational view
Implementation Example – LiteLLM Proxy Configuration:
For teams looking to implement a self-hosted LLM gateway, LiteLLM remains the most widely adopted open-source solution, supporting 100+ providers through an OpenAI-compatible API. Here’s a basic configuration:
config.yaml - LiteLLM Proxy Configuration model_list: - model_name: gpt-4 litellm_params: model: azure/gpt-4 api_base: https://your-azure-endpoint.openai.azure.com/ api_key: os.environ/AZURE_API_KEY - model_name: claude-3 litellm_params: model: anthropic/claude-3-opus-20240229 api_key: os.environ/ANTHROPIC_API_KEY - model_name: gemini-pro litellm_params: model: gemini/gemini-pro api_key: os.environ/GEMINI_API_KEY litellm_settings: drop_params: true set_verbose: true general_settings: master_key: os.environ/LITELLM_MASTER_KEY database_url: postgresql://user:pass@localhost:5432/litellm otel: true custom_auth: true
Deploy with Docker:
Pull and run LiteLLM proxy
docker run -d \
--1ame litellm-proxy \
-p 4000:4000 \
-v $(pwd)/config.yaml:/app/config.yaml \
-e LITELLM_MASTER_KEY="your-master-key" \
ghcr.io/berriai/litellm:main-latest \
--config /app/config.yaml --port 4000
Test the gateway
curl -X POST http://localhost:4000/chat/completions \
-H "Authorization: Bearer your-master-key" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4",
"messages": [{"role": "user", "content": "Hello, world!"}]
}'
Kubernetes Deployment with Helm:
Add Helm repository helm repo add litellm https://litellm.github.io/litellm-helm/ helm repo update Install with custom values helm upgrade --install litellm-proxy litellm/litellm-proxy \ -f values.yaml \ --1amespace ai-gateway \ --create-1amespace
values.yaml for Helm deployment replicaCount: 2 image: repository: ghcr.io/berriai/litellm tag: main-latest config: model_list: - model_name: gpt-4 litellm_params: model: azure/gpt-4 api_base: "https://your-endpoint.openai.azure.com/" general_settings: database_url: "postgresql://user:pass@postgres:5432/litellm" otel: true service: type: ClusterIP port: 4000 ingress: enabled: true className: nginx hosts: - host: ai-gateway.your-domain.com paths: - path: / pathType: Prefix
- MCP Gateway – Standardizing Tool Access Across AI Systems
The Model Context Protocol (MCP) is an open standard for connecting AI tools. But when every AI tool connects directly to MCP servers, you get MCP sprawl – the same problem as AI sprawl, now at the tool level. An MCP gateway serves as a centralized infrastructure component that transforms unmanaged AI integrations into a secure and scalable environment.
Glean MCP Gateway exposes tools to every assistant your teams use – Claude, ChatGPT, Cursor, Gemini, Microsoft Copilot, Claude Code, and more – while enforcing permissions, policy, and audit on every call. This includes search, read and write tools, custom tools, and any third-party MCP servers, all available via a single governed endpoint.
Key Capabilities:
- Tool Discovery and Aggregation: Automatically discovers and aggregates available tools from multiple MCP backends
- Tool Filtering: Control which tools are exposed using policy-based filtering
- Permission Enforcement: Enforces existing data permissions so models only operate on authorized data
- Audit Trails: Every MCP tool invocation is logged for security and compliance
MCP Gateway Implementation Example:
For teams building their own MCP gateway, the Envoy AI Gateway provides a robust foundation built on Kubernetes Gateway API:
Install Envoy AI Gateway with Helm helm repo add envoy-gateway https://gateway.envoyproxy.io/charts helm repo update helm install eg envoy-gateway/gateway-helm \ --1amespace envoy-gateway-system \ --create-1amespace Apply Gateway configuration kubectl apply -f - <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: ai-gateway spec: gatewayClassName: envoy-gateway-class listeners: - name: mcp port: 8080 protocol: HTTP allowedRoutes: namespaces: from: All EOF
MCP Server Registration:
MCP Server Configuration
apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: MCPBackend
metadata:
name: github-mcp
spec:
address: http://mcp-github-server:8080
type: github
authentication:
type: oauth2
clientId: ${GITHUB_CLIENT_ID}
clientSecret: ${GITHUB_CLIENT_SECRET}
apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: RoutePolicy
metadata:
name: tool-filtering
spec:
targetRef:
group: gateway.networking.k8s.io
kind: HTTPRoute
name: ai-route
toolSelector:
include:
- "github."
exclude:
- "github.delete."
Client Connection Example:
Python MCP Client connecting through gateway
import httpx
import json
Connect to MCP Gateway
response = httpx.post(
"http://ai-gateway:8080/mcp",
headers={"Authorization": "Bearer ${ACCESS_TOKEN}"},
json={
"method": "tools/list",
"params": {}
}
)
tools = response.json()
Call a discovered tool
tool_call = httpx.post(
"http://ai-gateway:8080/mcp",
headers={"Authorization": "Bearer ${ACCESS_TOKEN}"},
json={
"method": "tools/call",
"params": {
"name": "github.search_repositories",
"arguments": {"query": "langchain", "limit": 5}
}
}
)
print(tool_call.json())
- Governance That Doesn’t Reset with Every New Tool
Visibility and cost controls matter, but they don’t address the deeper question: whether AI is actually behaving consistently and safely across every front door. That requires governance that travels with the request rather than governance that has to be rebuilt every time a new front door appears.
Glean AI Gateway enforces existing data permissions so that models only ever operate on authorized data. The MCP Gateway extends enterprise-grade security to every external MCP call, with AI threat protection, granular access controls, compliance readiness for SOC 2 and ISO 27001, and authentication through OAuth 2.1 and enterprise IdPs.
Policy Enforcement Example:
Access Control Policy apiVersion: aigateway.envoyproxy.io/v1alpha1 kind: AIPolicy metadata: name: enterprise-governance spec: Rate limiting per user rateLimit: - type: USER limit: 1000 period: MINUTE Cost controls costControl: budgets: - name: engineering-budget limit: 5000 period: MONTH fallbackModel: llama-3-8b Security policies security: promptInjection: enabled: true action: BLOCK piiDetection: enabled: true action: REDACT Tool access controls toolAccess: - group: engineering tools: ["github.", "jira."] - group: finance tools: ["salesforce.", "stripe."] - default: ["search."]
- Observability – Knowing Where Tokens and MCP Tools Actually Go
Even with spend controls in place, most organizations are still in the dark when it comes to understanding which tasks are being tackled with AI. Knowing where tokens and MCP tools are actually going is what makes cost attribution, adoption tracking, performance monitoring, and threat detection possible.
Key Observability Metrics:
- Token Spend: Broken down by model, app, user, and provider
- MCP Tool Usage: Shows which capabilities are getting real traction and which use cases are ready to be scaled
- Model Performance: Latency and error rates in the same operational view as the rest of the AI stack
- Security Signals: Visibility into MCP tool invocations and access patterns gives security the signal it needs to spot anomalies before they become incidents
OpenTelemetry Integration:
OpenTelemetry integration for AI Gateway
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
Setup tracer
provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(<strong>name</strong>)
Instrument gateway requests
with tracer.start_as_current_span("llm_request") as span:
span.set_attribute("model", "gpt-4")
span.set_attribute("user", user_id)
span.set_attribute("app", app_name)
span.set_attribute("tokens_used", tokens_used)
span.set_attribute("cost", estimated_cost)
Make the LLM request
response = gateway.forward(request)
6. Enterprise Deployment Best Practices
Security Considerations:
- API Key Management: Provider keys live only in the gateway. Developers never see them
- Authentication: Use OAuth 2.1 and enterprise IdPs for all gateway access
- Network Security: Deploy gateway within your VPC with proper network policies
- Data Residency: Ensure model providers comply with data residency requirements
Cost Optimization Strategy:
- Match models to tasks – frontier models for complex reasoning, lighter models for high-volume work
- Implement budget alerts and automatic fallback
- Monitor token usage by team and project
Compliance Readiness:
- Maintain audit trails for every model interaction
- SOC 2 and ISO 27001 compliance readiness
- PII detection and redaction capabilities
What Undercode Say:
- Key Takeaway 1: AI sprawl is not just an inconvenience – it’s a governance, security, and cost crisis that compounds exponentially with every new tool adoption. Organizations without a centralized AI gateway will find themselves unable to control costs, enforce policies, or maintain visibility across their AI estate.
-
Key Takeaway 2: The separation of LLM Gateway (model access and routing) and MCP Gateway (tool access standardization) is architecturally sound. It allows organizations to govern both what models are used and what tools AI systems can access, all through a single control plane that doesn’t reset with every new AI front door.
Analysis: The Glean AI Gateway addresses a critical pain point that has been largely ignored in the rush to adopt AI tools. Most organizations have been operating under the illusion that AI adoption is simply about choosing the right tools and letting developers run with them. The reality is far messier – AI sprawl creates security blind spots, cost overruns, and governance gaps that traditional IT controls can’t address. The gateway pattern, already proven in API management, is the natural evolution for AI infrastructure.
What makes Glean’s approach particularly compelling is the MCP Gateway component. As AI agents become more autonomous, their ability to call external tools becomes both a superpower and a massive security risk. Standardizing tool access through a governed gateway – rather than letting each AI tool configure its own access – is essential for enterprise-grade deployments. The fact that Glean supports 30+ models and integrates with existing identity providers means organizations can adopt this without ripping and replacing their current AI stack.
However, organizations should consider that gateway solutions are not one-size-fits-all. Open-source alternatives like LiteLLM provide similar capabilities for teams with DevOps capacity, while managed solutions like Glean offer faster time-to-value. The choice depends on organizational maturity, compliance requirements, and available operational resources. What’s clear is that doing nothing is not an option – the AI sprawl problem will only get worse as more agents and tools enter the enterprise.
Prediction:
- +1 Enterprise AI gateway adoption will become mandatory for regulated industries within 18-24 months, driven by compliance requirements and audit demands for AI usage transparency.
-
+1 The MCP protocol will emerge as the de facto standard for AI tool integration, with MCP gateways becoming as ubiquitous as API gateways are today for traditional microservices.
-
-1 Organizations that fail to implement AI governance layers will face significant security incidents, cost overruns exceeding 300% of budget, and potential regulatory fines as AI usage becomes subject to increasing scrutiny.
-
+1 The AI gateway market will consolidate rapidly, with major cloud providers (AWS, Azure, GCP) building native AI gateway capabilities into their platforms, potentially commoditizing standalone solutions.
-
+1 Open-source AI gateways like LiteLLM will continue to thrive for development and testing, but enterprise deployments will increasingly favor managed solutions with built-in compliance, security, and support guarantees.
▶️ Related Video (76% Match):
https://www.youtube.com/watch?v=1YdbrpigINY
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
IT/Security Reporter URL:
Reported By: Sumanth077 Managing – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
