Protect Kubernetes Pods with Resource Limits

A single container without a memory limit can consume every byte on its node and take unrelated Pods down with it. Kubernetes requests and limits prevent that, but the wrong numbers create their own failures: OOMKilled restart loops, CPU throttling under load, and Pods stuck in Pending. This tutorial shows how to set them, enforce them per namespace, and verify the result.

What are Kubernetes resource limits?

Kubernetes resource limits are per-container upper bounds on CPU and memory that the kubelet enforces through Linux cgroups: a container above its memory limit is killed with exit code 137, a container above its CPU limit is throttled instead of killed.

Requests serve a different purpose. The scheduler looks only at requests when it decides which node a Pod fits on, and it never looks at limits. A Pod requesting 2 CPUs is placed on a node with 2 free CPUs. A Pod requesting nothing can be placed on an already saturated node.

Prerequisites

• A Kubernetes cluster, version 1.28 or newer, for example a managed Kubernetes cluster with a free control plane
• kubectl configured against that cluster, with permission to create namespaces and quotas
• metrics-server deployed in the cluster, required for kubectl top
• A scratch namespace: kubectl create namespace limits-demo

How does a container behave when it hits its limit?

CPU and memory behave differently: CPU is compressible, so the kernel throttles the container through its CFS quota, while memory is incompressible, so the kernel OOM killer terminates the process immediately.

graph TD
  A["Container requests more resources"] --> B{"Resource type"}
  B -->|CPU| C{"Above cpu limit?"}
  C -->|yes| D["CFS throttling, container keeps running"]
  C -->|no| E["Container runs at full speed"]
  B -->|Memory| F{"Above memory limit?"}
  F -->|yes| G["OOMKilled, exit code 137, restart per policy"]
  F -->|no| H{"Node under memory pressure?"}
  H -->|yes| I["kubelet evicts by QoS class"]
  H -->|no| J["Allocation succeeds"]

That asymmetry drives two practical rules. Memory limits must be generous enough to absorb legitimate peaks, because the penalty for being wrong is a killed process. CPU limits are optional for many latency-sensitive workloads, because a tight CPU limit adds latency without protecting anything the CPU request does not already protect.

Set requests and limits on a Pod

Resources are declared per container under spec.containers[].resources. The following Deployment sets both values for an HTTP service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: limits-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: ghcr.io/YOUR_ORG/api:1.4.2
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "1"
              memory: "512Mi"

Apply it and watch the rollout:

$ kubectl apply -f /root/manifests/api-deployment.yaml
$ kubectl -n limits-demo rollout status deployment/api

The units matter. CPU is measured in cores, where 1 is one full core and 250m is a quarter of a core. Memory accepts binary suffixes (Ki, Mi, Gi) and decimal suffixes (K, M, G). 1Gi is 1,073,741,824 bytes, 1G is 1,000,000,000 bytes. Use the binary form so the number matches what your monitoring reports.

Runtimes with their own heap management need the limit passed through. A JVM container with memory: "512Mi" should run with -XX:MaxRAMPercentage=75, otherwise the heap grows past the cgroup limit and the JVM is killed rather than throwing an OutOfMemoryError.

What are the three QoS classes?

Kubernetes assigns every Pod exactly one QoS class: Guaranteed when requests equal limits for CPU and memory in every container, Burstable when at least one request or limit is set but they are not equal, and BestEffort when nothing is set.

The class is derived, not configured. You cannot set it in the manifest. For stateful workloads and anything that must survive a noisy node, set requests equal to limits and accept the lower packing density.

Measure real usage before choosing numbers

Guessed limits are the root cause of most OOM kills. Collect actual consumption over a representative period first:

$ kubectl -n limits-demo top pod --containers
POD                    NAME   CPU(cores)   MEMORY(bytes)
api-7d9c5f8b96-2xk4n   api    180m         301Mi
api-7d9c5f8b96-8vlqp   api    212m         288Mi

Set the memory request near the observed steady state and the memory limit roughly 50 percent above the observed peak. Set the CPU request at the median and leave the CPU limit off unless you have a concrete reason, such as a batch job that must not starve co-located services.

Then check whether the nodes can carry the sum of all requests:

$ kubectl describe node <node-name> | grep -A 6 "Allocated resources"

If the requested percentages on every node are already above 80 percent, tuning individual Pods will not create room. Add capacity instead, for example additional worker nodes on scalable Linux VMs with shared or dedicated vCPUs, and keep enough headroom for one node to fail without leaving Pods unschedulable.

Enforce defaults with a LimitRange

A LimitRange injects requests and limits into containers that declare none, and rejects containers whose values fall outside the allowed range:

apiVersion: v1
kind: LimitRange
metadata:
  name: container-defaults
  namespace: limits-demo
spec:
  limits:
    - type: Container
      default:
        cpu: "500m"
        memory: "512Mi"
      defaultRequest:
        cpu: "100m"
        memory: "128Mi"
      max:
        cpu: "2"
        memory: "4Gi"
      min:
        cpu: "50m"
        memory: "64Mi"
      maxLimitRequestRatio:
        memory: "4"

• default sets the limit for containers without one.
• defaultRequest sets the request for containers without one.
• max and min reject manifests outside the range with a validation error at admission time.
• maxLimitRequestRatio blocks extreme overcommit, here a memory limit more than four times the request.

A LimitRange only applies to Pods created after it exists. Restart existing workloads to pick up the defaults:

$ kubectl apply -f /root/manifests/limitrange.yaml
$ kubectl -n limits-demo rollout restart deployment/api

Cap a namespace with a ResourceQuota

A LimitRange bounds a single container, a ResourceQuota bounds the namespace total:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: limits-demo
spec:
  hard:
    requests.cpu: "8"
    requests.memory: "16Gi"
    limits.cpu: "16"
    limits.memory: "32Gi"
    pods: "40"

Once a quota tracks requests.cpu or limits.memory, every new container in that namespace must declare the matching value or creation fails with must specify limits.memory. This is why the LimitRange should be applied first: it supplies the values that the quota then demands.

Verification

Confirm the QoS class the API server assigned:

$ kubectl -n limits-demo get pod -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass
NAME                   QOS
api-6b8f4d7c55-4rk9t   Burstable

Confirm the limit reached the container's cgroup. On a cgroup v2 node the value is in bytes, 536870912 for 512Mi:

$ kubectl -n limits-demo exec deploy/api -- cat /sys/fs/cgroup/memory.max
536870912

Confirm the quota accounting:

$ kubectl -n limits-demo describe resourcequota team-quota
Name:            team-quota
Resource         Used    Hard
--------         ----    ----
limits.memory    1536Mi  32Gi
pods             3       40
requests.cpu     750m    8

Troubleshooting

Pod restarts with OOMKilled

Check the previous termination reason rather than the current state:

$ kubectl -n limits-demo get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}'
OOMKilled

The container exceeded limits.memory. Raise the limit to about 1.5 times the observed peak, or fix the leak. Raising the request without raising the limit changes nothing, because the limit is what the OOM killer enforces.

Pod stays in Pending

$ kubectl -n limits-demo describe pod <pod-name> | tail -5

A message such as 0/3 nodes are available: 3 Insufficient cpu means no node has enough unreserved CPU for the request. Lower the request to a value backed by measurement, or add a node. Note that the scheduler counts requests of all Pods on the node, not their current usage, so a node can be idle and still reject the Pod.

Latency spikes although CPU usage looks low

Average utilisation hides CFS throttling. Query the counter directly:

$ kubectl -n limits-demo exec deploy/api -- cat /sys/fs/cgroup/cpu.stat | grep throttled
nr_throttled 4821
throttled_usec 39114522

A rising nr_throttled means the container hit its CPU quota inside a 100 ms scheduling period, even if the one-minute average sits at 20 percent. Raise limits.cpu or remove it and rely on the request for share allocation.

Next steps

Apply the pattern in this order: measure with kubectl top, set explicit requests and limits per container, add a LimitRange so nothing lands unbounded, then a ResourceQuota per team namespace. Review the numbers after each significant release, since a changed heap setting or a new cache invalidates yesterday's measurement. For workloads whose demand shifts over time, look at the Vertical Pod Autoscaler in recommendation mode, and at the in-place Pod resize feature, which reached beta in Kubernetes 1.33 and allows changing CPU and memory without recreating the Pod.