Kubernetes is the operating system of modern cloud infrastructure, and it is genuinely harder to defend than the environments most security teams grew up on. Expel’s March 2026 conversation with Ethan Chen lays out the core reasons well: dynamic and ephemeral infrastructure, a Russian-doll layering problem, an enormous communication surface, and workloads that disappear before an investigation can attribute behaviour to them. This post takes that framing and extends it into the mechanics: what the challenges look like at the layer of concrete API calls, ports and misconfigurations, what changed in 2025 and 2026, and what detection and hardening look like when you have to do this for real.
The one-paragraph version
Kubernetes security is hard not because the individual components are exotic, but because the platform concentrates identity, network reachability and workload lifecycle decisions in configuration that no single team fully owns. Almost every public incident traces back to a misconfiguration or an over-broad identity rather than a novel exploit: exposed node agents, permissive RBAC, workload credentials that reach cloud accounts, images pulled by mutable tag, and an ingress layer that just lost its maintainers. Surveys reflect this: in Red Hat’s 2026 state of cloud-native security reporting, 97% of surveyed organizations reported at least one cloud-native security issue in the past year, 90% reported a Kubernetes security incident, and misconfiguration is repeatedly identified as the largest single driver (compiled by StationX from Red Hat’s survey data).
Why Kubernetes resists the usual security model
Expel’s assessment names the symptoms. Here is the underlying mechanism for each, plus what it breaks in day-to-day defence.
1. Identity becomes the only stable thing, and most clusters hand it out for free
In a server world you could segment by IP, track an asset by hostname and reason about a machine over months. In Kubernetes, pods are scheduled anywhere, pod IPs change on restart, and the same workload may run on three different nodes in a week. The only element that persists across those moves is identity: the service account, the workload identity, the cloud role behind it. That is why the service account token matters more than any single CVE. Every pod gets one mounted at /var/run/secrets/kubernetes.io/serviceaccount/token by default, along with the cluster CA and namespace, and it is a bearer credential that any process in that container can read. Attackers do not need to exploit your application if they can read the filesystem: the token is already there.
The defensive consequence is blunt: treat every pod as holding a credential, disable token automount where it is not needed, bound tokens to audiences and short lifetimes, and audit what each service account can actually do rather than what someone intended it to do.
2. The Russian doll: blast radius cascades across layers you do not own
Cluster, node, pod, container: a weakness at any layer can compromise the one below it, and the escalation paths between layers are the interesting part. A privileged pod can mount the node filesystem. A hostPath volume to /var/run/docker.sock or / is a container escape waiting to be used. hostPID lets a container see and signal host processes. A container runtime bug such as CVE-2024-21626 in runc (the ‘Leaky Vessels’ disclosure: a leaked file descriptor allowed the container process to land in a host directory) turns ordinary container start into a host compromise. Once an attacker owns the node, they own every pod scheduled on it and often the node’s cloud instance role too.
This is the layer where teams most often have a blind spot, because the control plane is owned by the provider (in managed Kubernetes) and the node is owned by nobody in particular.
3. Networking is default-permissive, and policy is opt-in
Without NetworkPolicy objects, every pod can reach every other pod in the cluster, and egress is typically unrestricted. There is no implicit segmentation to fall back on. Service discovery makes this worse in a useful way for attackers: DNS names are predictable, and internal services are reachable by name from any compromised container. In practice this means the first foothold rarely needs lateral movement tooling; it just talks to the next service by its cluster DNS name.
Default-deny policy for both ingress and egress, applied namespace by namespace, is one of the highest-value changes available. It is also one of the most frequently skipped because it requires knowing legitimate traffic patterns first.
4. The API server is a root-equivalent service, and RBAC is subtler than it looks
Everything in Kubernetes is a request to the API server, which makes authorization the real security boundary. RBAC looks simple (verbs, resources, subjects) and is easy to get wrong: wildcard resources and verbs, cluster-admin bound to a default service account, roles that allow create on pods (which permits mounting any of those pods’ volume types), get on secrets across all namespaces, or the escalate and bind verbs that let a subject grant themselves new permissions. There are also non-obvious paths: approving certificate signing requests grants cluster identities, and creating a mutating admission webhook gives durable, cluster-wide code execution inside the control plane path.
Practical audit: list every subject with cluster-admin, find wildcards, check who can read secrets, exec into pods or create workloads, and treat service accounts with create-on-pods as effectively privileged.
5. Shared responsibility turns into shared confusion
In managed offerings (EKS, GKE, AKS), the provider secures the control plane; you secure nodes, workloads, RBAC, network policy, secrets and the integration with cloud identity. The seam between the two is where incidents live. The mapping of a pod service account to cloud permissions, through IAM Roles for Service Accounts on AWS or Workload Identity on GCP, is a genuinely elegant design and a genuinely dangerous one: an over-scoped trust policy turns a compromised pod into an S3-reading, secret-manager-reading, sometimes role-assuming cloud principal. SentinelOne’s reporting that the majority of cloud intrusions originate from compromised identities fits this model exactly.
6. Supply chain, and the ingress cliff that arrived in March 2026
Images pulled by mutable tag, Helm charts installed from public repositories, operators shipped with cluster-admin, and CI pipelines holding registry push rights: all of these are inside the trust boundary of the cluster even though the code came from outside it. Admission webhooks add a second-order risk, since a webhook is code the API server will call for every matching request.
And then there is the sharpest change since Expel’s conversation: Ingress NGINX was retired in March 2026. Kubernetes SIG Network and the Security Response Committee announced it in November 2025, with maintenance ending in March 2026 and no further releases, bug fixes or security patches afterwards. According to Datadog research cited in the Kubernetes announcement, roughly half of cloud-native environments still rely on it. Adding the IngressNightmare vulnerability family (CVE-2025-1974 plus CVE-2025-1097, CVE-2025-1098 and CVE-2025-24514, CVSS 9.8, unauthenticated RCE reachable from the pod network through the admission webhook, with Wiz estimating about 43% of cloud environments exposed), the operational conclusion is unavoidable: any remaining Ingress NGINX deployment is both historically vulnerable and permanently unpatched, which converts a configuration problem into an architectural, time-boxed migration project.
Figure 1. Four common entry points, five pivots, and the impact set. The pivots, not the exploits, decide blast radius.
The attack surface, concretely
Generic advice improves when it is attached to ports, paths and flags you can actually check. This is the surface worth inventorying in any cluster.
| Component | Typical exposure | What an attacker gains | First control |
|---|---|---|---|
| API server | Port 6443, public endpoint in unhardened setups | Full cluster control if anonymous or weakly authorized; everything else is a detail | Private endpoint, no anonymous auth, strong authn plus audit logging |
| kubelet | 10250 (authenticated API), 10255 (read-only, legacy), often reachable from the pod network | List pods, read logs, exec commands inside containers | Disable anonymous auth, restrict network reachability, rotate node credentials |
| etcd | 2379/2380 if exposed or unencrypted | Every Secret and object in the cluster, in plaintext | Provider-managed or mTLS only, never routable, encryption at rest |
| Kubernetes Dashboard | Often deployed then forgotten, weak or anonymous access | Interactive cluster access with the dashboard’s own permissions | Remove if unused, otherwise SSO plus least-privilege role |
| Ingress controller | Retired Ingress NGINX; alternative controllers with their own CVEs | Reachability from the internet into the admission path (CVE-2025-1974) or into applications | Migrate to a maintained controller and Gateway API, patch aggressively |
| Pod network | Flat by default, no policy required to reach neighbours | Lateral movement without tooling, database and internal API access | Default-deny NetworkPolicy for ingress and egress |
| Service account token | Mounted into every pod unless disabled | Bearer credential usable against the API server | automountServiceAccountToken false, bound audiences, short lifetimes |
| Cloud identity bridge | IRSA or Workload Identity trust policies | Cloud API access: storage, secrets, databases, sometimes role assumption | Scope trust to specific subjects and actions, review assumed-role chains |
| Container runtime and node OS | runc, containerd, kernel and node AMI patch levels | Host escape (CVE-2024-21626 class), then every pod on the node | Latest runc and node AMIs, privileged workload ban, seccomp RuntimeDefault |
| Registry and CI | Mutable tags, public base images, push credentials in CI | Malicious image pulled by tag drift, supply-chain execution | Digest pinning, signature verification, registry allowlist, CI hardening |
Case study: from an exposed kubelet to a cluster-wide miner
The Hildegard campaign (Unit 42, January 2021, attributed to TeamTNT) remains the cleanest worked example of how a misconfigured node agent becomes cluster-wide abuse, and the pattern is still alive because the misconfiguration is still common. The kubelet is the node agent that manages pods; when anonymous authentication or permissive authorization is enabled and the port is reachable, it becomes an execution API that does not require the cluster’s authorization layer at all.
Figure 2. The Hildegard pattern: recon, execution, internal scanning, mining, evasion. No API server access required.
- Recon. Internet-wide scanning for port 10250 (and legacy 10255), the same way Shodan and masscan find any exposed service.
- Initial access. Anonymous requests against the kubelet: no exploit, just a permissive configuration in a cluster the operator thought was private.
- Survey. The attacker enumerates pods through the kubelet and chooses a container to work from.
- Execution. The kubelet’s exec endpoints provide a command channel inside the running container.
- Command and control. The attacker downloads tmate and opens a reverse session to tmate.io, giving an interactive shell that survives NAT and outbound rules written with only HTTP in mind. Hildegard also used an IRC channel as a second C2 path.
- Internal reconnaissance. From inside the pod network, masscan sweeps the cluster ranges for additional unsecured kubelets on other nodes.
- Propagation. The same playbook repeats node by node: one misconfigured node becomes the whole cluster.
- Monetization. A shell script (xmr.sh in the Hildegard samples) downloads and runs a Monero miner pointed at the attacker’s wallet.
- Evasion. The miner runs under a kernel-sounding process name (bioset in that campaign), and the malware used LD_PRELOAD library injection to hide itself, with additional persistence so mining resumes across container restarts.
Two properties of that chain are worth internalizing. First, it never touches the API server, so controls focused exclusively on API access are blind to it. Second, the observable outcome is mundane: a process with an unexpected name, a network connection to a mining pool, a CPU profile that looks like a busy production workload. Detection here is behavioural, not signature-based.
Case study: IngressNightmare and the retirement cliff
IngressNightmare is the strongest recent illustration of the admission-path risk. Wiz Research disclosed four CVEs in the Ingress NGINX controller in March 2025: CVE-2025-1097, CVE-2025-1098, CVE-2025-24514 and the 9.8-rated CVE-2025-1974. The mechanics matter: the controller’s admission webhook is reachable from the pod network without authentication, and by feeding it crafted Ingress objects, an attacker can inject NGINX configuration directives that result in remote code execution in the controller pod. That pod is not just a web front door. It holds a service account that can read Secrets across namespaces, which is why the disclosure stresses cluster takeover rather than a single application compromise.
- Exposure: Wiz estimated roughly 43% of cloud environments were vulnerable at disclosure, and the Kubernetes retirement notice cites Datadog data that about half of cloud-native environments run Ingress NGINX.
- Detection angles: unexpected requests to the ingress admission webhook originating inside the cluster, Ingress objects with unusual annotation or configuration directives, configuration reloads and restarts of the controller outside deployment windows, and subsequent bursts of Secret reads.
- Remediation, in order: upgrade to the last fixed releases wherever it still exists, inventory every cluster and ingress class so nothing is forgotten, then migrate to a maintained controller (Gateway API implementations or a vendor ingress), and remove the retired controller entirely rather than leaving it running unpatched.
Case study: escapes and why node compromise ends the argument
Container escapes are comparatively rare, and when they happen they are decisive. The ‘Leaky Vessels’ set disclosed in January 2024 is the reference example: CVE-2024-21626 in runc let a container process inherit a file descriptor to a host directory (via the leaked descriptor and process working directory handling), so a crafted image or RUN instruction could land the process in the host filesystem, with related BuildKit issues (CVE-2024-23651, CVE-2024-23652, CVE-2024-23653) affecting image builds. Patched runc and BuildKit versions shipped quickly, which is exactly the point: the control that matters is not clever detection, it is staying current on the components that implement isolation.
The pre-packaged variants are usually easier for attackers than a CVE. A privileged pod is a host escape by design: mount the host filesystem, read the kubelet’s credentials, or use the node’s cloud instance role. hostPath volumes to / or to the container runtime socket, hostNetwork, hostPID and CAP_SYS_ADMIN are all one decision away from node compromise, which is why Pod Security Standards place them behind the restricted profile.
Detection engineering: what to log, and what actually signals an intrusion
Kubernetes gives defenders telemetry that most organizations under-collect. The programme below is ordered by return on effort.
Telemetry you must have
- API audit logs at RequestResponse level for the sensitive resources: pods and pods/exec, secrets, rolebindings and clusterrolebindings, serviceaccounts and tokens, validatingwebhookconfigurations and mutatingwebhookconfigurations, certificatesigningrequests, and nodes. Ship them off-cluster; the cluster cannot be trusted to store evidence about itself.
- Kubelet logs plus node OS auditd where available, because the Hildegard chain is only visible at the node agent.
- Cloud control-plane logs (EKS audit to CloudWatch, GKE/AKS equivalents) and cloud identity logs for the assume-role or token exchange path behind workload identity.
- Runtime telemetry from eBPF tooling such as Falco or Tetragon: process execution, shell spawns, file writes, mount and namespace syscalls, and outbound connection attempts.
- Registry and pipeline logs: which images were pushed, by whom, with what digest, and what the cluster pulled.
- Network flow data for egress, which is where resource hijacking becomes obvious.
High-signal behaviours
| Signal | Why it matters | Where it appears |
|---|---|---|
| pods/exec or pods/attach by an unexpected user agent | Interactive access to a running container is how most hands-on intrusions proceed | API audit log |
| Privileged pod, hostPath to / or the runtime socket, hostPID, hostNetwork | Pre-packaged host escape and node compromise | API audit log, admission webhook decisions |
| New rolebinding to cluster-admin, or wildcard roles created | Privilege escalation to the whole cluster | API audit log |
| Burst of Secret reads or list/watch across namespaces | Collection phase of a cluster takeover | API audit log |
| Creation or modification of admission webhook configurations | Durable execution inside the control plane path | API audit log |
| Kubelet API calls from outside the node or outside the cluster CIDR | Anonymous access abuse, the Hildegard entry point | Kubelet logs, network flow |
| Exec of a shell in a workload that normally has none | Post-exploitation inside a container | Runtime eBPF |
| Downloads of tmate, masscan, or shell scripts from unknown hosts | Established TeamTNT-class tooling chain | Runtime eBPF, network flow, file integrity |
| Outbound traffic to mining pools or IRC, or sustained CPU with no business traffic | Resource hijacking | Network flow, cloud billing anomalies |
| IMDS queries or cloud token exchange from unexpected pods | Identity pivot from cluster to cloud account | Cloud trail, runtime eBPF |
These map cleanly onto MITRE ATT&CK for Containers: T1610 Deploy Container, T1611 Escape to Host, T1613 Container and Resource Discovery, T1609 Container Administration Command, T1552.007 unsecured credentials from the container API, T1550.001 application access tokens, T1496 Resource Hijacking, T1078.004 cloud accounts, T1526 cloud service discovery, and T1195 supply-chain compromise.
Two snippets show what the rules look like in practice. The first is the audit filter for interactive access; the second is a runtime rule for the miner pattern.
# Audit-log filter: interactive access and privilege changes (pseudo-syntax)
# fields: verb, resource, subresource, user.username, userAgent, responseStatus.code
(verb in (create) and resource in (pods) and subresource in (exec, attach, portforward))
OR (verb in (create, update) and resource in (clusterrolebindings, rolebindings)
and requestObject.roleRef.name == "cluster-admin")
OR (verb in (list, watch, get) and resource in (secrets) and namespace != "kube-system")
# alert on: unexpected userAgent (kubectl, curl, python-requests) seen from workload IPs,
# and on any of the above from a service account rather than a human identity.# Falco-style runtime rule: resource hijacking behaviour
- rule: Miner or reverse-shell tooling started in container
desc: Known cryptojacking and C2 tooling executed inside a workload
condition: >
spawned_process and container and
(proc.name in (xmrig, xmr.sh, masscan, tmate, tmate-ssh-server) or
(proc.name in (bioset, kworker, ksoftirqd) and proc.pcmdline contains "/tmp") or
proc.cmdline contains "stratum+tcp://")
output: "Suspicious process in container (proc=%proc.name cmd=%proc.cmdline container=%container.name image=%container.image.repository)"
priority: CRITICALOne caveat worth stating plainly: detection in Kubernetes is only as good as the identity context attached to events. If your audit logs do not show which service account, which namespace and which workload generated a request, you will spend your investigation budget reconstructing what the cluster already knew.
Hardening playbook, prioritized
Figure 3. The eight control layers, and what each one actually catches. Every layer must ship telemetry or the next hop is invisible.
P0: this week
- Inventory every cluster and who can reach it: API endpoints, kubelets, dashboards, ingress controllers. Look specifically for kubelet 10250/10255 exposure from outside the node network.
- Turn on API audit logging at RequestResponse for the sensitive resources and ship it off-cluster. Without this, nothing below can be verified or investigated.
- Remove anonymous access: kubelet anonymous auth disabled, no anonymous API or dashboard access, no cluster-admin bound to default service accounts.
- Enforce Pod Security Standards at restricted for namespaces that can take it, and audit the rest so you know the exceptions you are choosing.
- Find and schedule the Ingress NGINX migration, and patch any remaining deployment to its last fixed release today.
P1: this month
- RBAC review with tooling for what roles can actually do, not what they say: eliminate wildcards, remove unused bindings, replace cluster-admin with scoped roles, and set automountServiceAccountToken false where tokens are not needed.
- Default-deny NetworkPolicy for ingress and egress on the namespaces that handle the most sensitive data, with explicit allowances for real traffic.
- Scope workload identity: review IRSA or Workload Identity trust policies, remove wildcard actions and resources, and confirm which pods can reach cloud storage or secrets managers.
- Runtime detection deployed and tuned: Falco or Tetragon with rules for exec, shell spawns, mount syscalls, miner behaviour and outbound tooling, wired into the SIEM with alerts a human can act on.
- Image discipline: pin by digest, verify signatures with cosign where possible, allowlist registries, and scan in the pipeline rather than in production.
P2: this quarter
- Secrets programme: external secret manager with a CSI driver, etcd encryption at rest verified, rotation of long-lived credentials, and audit trails on who read which secret.
- Admission policy as code: Kyverno or Gatekeeper policies in Git for image provenance, required labels, resource limits, and banned pod specs, reviewed like application code.
- Node lifecycle: patch cadence for node AMIs, runc and containerd versions, and a documented process for draining and replacing nodes.
- Rehearsed Kubernetes incident response: cordon and drain rather than delete, capture the pod spec, image digest and node state, use ephemeral containers for triage instead of trusting a compromised image, rotate service account tokens and cloud credentials, and check identity trust policies for persistence.
| Managed platform | Provider owns | You own (in practice) |
|---|---|---|
| EKS / GKE / AKS | Control plane availability, control plane patching, etcd, API endpoint protection | Nodes and their OS, RBAC, network policy, workload security, secrets, workload identity trust, audit log configuration and retention, node patch cadence |
| Self-managed | Nothing | Everything, including etcd, control plane upgrades and API server flags, which is why the CIS Kubernetes Benchmark and the NSA/CISA hardening guidance exist |
What this adds to Expel’s framing
- The ingress cliff is new. The challenges Expel describes are structural, but the March 2026 Ingress NGINX retirement converts one of them, controller supply-chain and patching, into a dated migration obligation that no amount of detection compensates for.
- Identity, not network, is the perimeter. Ephemerality destroys asset-centric security, so RBAC and workload identity are not controls among others; they decide the blast radius of every other failure.
- Detection needs two planes at once. Audit logs explain what the cluster was asked to do; runtime telemetry explains what the processes actually did. Teams that instrument only one of the two planes will keep discovering intrusions from the cloud billing page.
Sources and further reading
- Expel: Why is Kubernetes security so challenging? conversation with Ethan Chen, March 2026: expel.com/resource/why-is-kubernetes-security-so-challenging
- Wiz Research: IngressNightmare, CVE-2025-1974 and related CVEs in Ingress NGINX: wiz.io/blog/ingress-nginx-kubernetes-vulnerabilities
- Kubernetes: Ingress NGINX retirement announcement (November 2025) and statement from SIG Network and the Security Response Committee (January 2026): kubernetes.io/blog/2025/11/11/ingress-nginx-retirement, kubernetes.io/blog/2026/01/29/ingress-nginx-statement
- Unit 42: Hildegard, TeamTNT cryptojacking malware targeting Kubernetes: unit42.paloaltonetworks.com/hildegard-malware-teamtnt
- MITRE ATT&CK for Containers: attack.mitre.org/matrices/enterprise/containers
- Palo Alto Networks: Leaky Vessels container escape vulnerabilities (CVE-2024-21626, CVE-2024-23651/52/53): paloaltonetworks.com/blog/cloud-security/leaky-vessels-vulnerabilities-container-escape
- Snyk Labs: CVE-2024-21626 runc process working directory and leaked file descriptors: labs.snyk.io
- Kubernetes documentation: kubelet configuration reference (anonymous authentication, authorization modes): kubernetes.io/docs/reference/command-line-tools-reference/kubelet
- NSA/CISA: Kubernetes Hardening Guidance: media.defense.gov (CTR_KUBERNETES_HARDENING_GUIDANCE)
- Falco and Tetragon documentation for runtime rules and eBPF enforcement: falco.org, tetragon.io
- Red Hat: The state of cloud-native security, 2026 edition, and the StationX compilation of cloud security statistics: redhat.com, app.stationx.net/articles/cloud-security-statistics
Published for defensive education. Configuration defaults and version behaviour change between Kubernetes releases and managed offerings; verify current guidance against your platform’s documentation before applying controls, and validate any indicator against your own telemetry before alerting on it.