Cgroups: A Deep Dive into Resource Management in Kubernetes – Part 3: Memory

In Part 1 we saw that Kubernetes writes your resources: block into cgroup files that the kernel enforces. In Part 2 we saw how CPU limits make a container wait, sometimes badly enough to fail its health checks.

Memory is different. The kernel can't make a program wait for memory it already holds, so when memory runs out, something has to die. This part answers the other promise from the start of the series: why a pod can be OOM-killed while the dashboard says it had room to spare.

In this part:

• Memory limits: OOM kills, and why the dashboard said there was room.
• Node pressure: what happens when the whole node runs short.
• In production: the memory numbers to watch, and the ones that lie.
• Beyond CPU and memory: the other limits, and a finale where you resize a running pod.

To follow along, set up the lab from Part 1 first.

#Memory limits

CPU can make you wait. Memory can't (chapter 6), so it fails in a completely different way.

#16. Memory limits and the OOM killer

Question: what happens when a container reaches memory.max?

Kubernetes didn't kill your OOMKilled pod. The Linux kernel did, because of one number in one file: memory.max.

If a program needs more memory and its box is full, waiting doesn't help.

When a cgroup reaches memory.max, the kernel first tries to reclaim: it frees memory it can safely drop, such as cached file data. Only if that isn't enough does it call the OOM killer, which picks a process inside the cgroup and ends it.

Try it yourself. memory-hog has a 64Mi limit and runs a small Python script that grabs 10 MiB every 0.3 seconds and never lets go:

kubectl -n lab apply -f labs/16-memory-limits-and-oom/pod.yaml
kubectl -n lab get pod memory-hog -w

Within seconds the status flips to OOMKilled, Kubernetes restarts it, and it happens again. Press Ctrl-C and look at what it printed before it died:

kubectl -n lab logs memory-hog --previous | tail -3

allocated 40 MiB
allocated 50 MiB
allocated 60 MiB

It never reached 64. The Python interpreter itself lives in the same box and uses memory too.

Kubernetes records the death as reason: OOMKilled, exitCode: 137. Exit code 137 means "killed by a signal": 128 plus the signal number, and signal 9 is SIGKILL, the one a program can't catch or ignore. It gets no chance to clean up or write a last log line.

The kernel keeps its own tally in the pod's memory.events:

POD_DIR=$(./scripts/pod-cgroup.sh memory-hog)
docker exec under-the-hood-worker cat "$POD_DIR/memory.events"

low 0
high 0
max 75
oom 2
oom_kill 4
oom_group_kill 2

oom_group_kill shows up because, on cgroups v2, the kubelet sets memory.oom.group = 1 on each container by default (since Kubernetes 1.28): if one process has to die, the whole container goes, so nothing is left half-alive.

Lab: 16-memory-limits-and-oom. Check what you built with make check LAB=16.

#17. Not all memory is equal

Question: how can a pod be OOM-killed while the dashboard said it had room to spare?

This answers the promise from the start of the series. The key is that a cgroup's memory comes in two kinds.

Both count towards memory.current and towards memory.max. But only one of them can get you killed.

Try it yourself. mem-lab has a 128Mi limit. First, read 192 MiB of files through it. The lab's cache command writes the file in 16 MiB steps, then reads it back twice:

kubectl -n lab apply -f labs/17-not-all-memory-is-equal/pod.yaml
kubectl -n lab wait --for=condition=Ready pod/mem-lab
kubectl -n lab exec mem-lab -- cache --mib 192
kubectl -n lab exec mem-lab -- cat /sys/fs/cgroup/memory.current
kubectl -n lab exec mem-lab -- grep -E '^(anon|file|inactive_file) ' /sys/fs/cgroup/memory.stat

On my run, memory.current read 125 MiB, almost exactly the limit, of which 122 MiB was file and under 1 MiB was anon. The pod's memory.events showed max 1814 (the limit was hit over and over) and oom_kill 0. Every time, the kernel dropped some old file cache and carried on. The container is "at its limit", and perfectly fine.

Now allocate memory that can't be dropped. The lab's alloc command grabs memory in steps and never lets go:

kubectl -n lab exec mem-lab -- alloc --mib 16 --every 100ms
kubectl -n lab get pod mem-lab

This time it ends with exit code 137 and a container restart, and memory.events moves to oom 1, oom_group_kill 1. There was no cache left to drop, only memory a running program was holding, so the only way to stay under the limit was to kill it.

#The working set: the number that matters

Because file cache inflates memory.current, the kubelet and most dashboards use the working set instead:

working set = memory.current − inactive_file

inactive_file is file cache that hasn't been touched recently, the first thing the kernel drops. The working set leaves it out, so it's a much better guess at what the container really needs. In the run above: memory.current 125 MiB, working set 1 MiB. Same container, same moment, two completely different stories. It's the number behind kubectl top and the metric container_memory_working_set_bytes.

#So why did the dashboard say there was room?

Usually one of three reasons, and now you can tell them apart:

1. A short spike. Most dashboards sample memory every 15 to 60 seconds, but the kernel reacts as soon as usage reaches memory.max. A spike between two samples never shows on the graph.
2. The wrong denominator. Many panels show memory as a share of the request or of the node, not of the limit.
3. The wrong kind of memory. A graph of memory.current mixes in reclaimable cache; a graph of the working set doesn't. They can tell very different stories.

memory.events doesn't miss anything, so when in doubt, trust it over the graph.

Lab: 17-not-all-memory-is-equal. Check what you built with make check LAB=17.

#Node pressure

#18. When the whole node runs short

Question: who gets removed when a node runs low on memory?

The other way to run out is for the whole node to get low on memory, because many pods use more than they requested. Three mechanisms decide what happens then.

#Not all of a node is for pods

A node can't hand all its memory to pods; the operating system and Kubernetes' own programs need some too. What's left is called allocatable:

kubectl get node under-the-hood-worker \
  -o jsonpath='{.status.capacity.memory}{"  capacity\n"}{.status.allocatable.memory}{"  allocatable\n"}'

8126480Ki  capacity
8126480Ki  allocatable

They're equal on kind because it reserves nothing. On GKE or EKS, several hundred MiB or more are held back through three kubelet settings: systemReserved (for the OS), kubeReserved (for the kubelet and container runtime) and evictionHard (a safety margin). The scheduler only ever compares pod requests with allocatable, never real usage, and the kubelet writes capacity − systemReserved − kubeReserved into the memory.max of the box for all pods. That's the 8321515520 from chapter 11 (exactly 8,126,480 Ki).

#QoS classes: who is most protected

Kubernetes sorts every pod into one of three Quality of Service classes, worked out from its requests and limits:

The class shows up in two places. In the tree, each class gets its own box (chapter 11). And on every process, as a number the kernel reads when the whole machine is out of memory:

kubectl -n lab apply -f labs/11-the-cgroup-tree/pods.yaml
kubectl -n lab wait --for=condition=Ready pod/qos-guaranteed pod/qos-burstable pod/qos-besteffort
for p in qos-guaranteed qos-burstable qos-besteffort; do
  echo "$p $(kubectl -n lab exec $p -- cat /proc/1/oom_score_adj)"
done

qos-guaranteed -997
qos-burstable 996
qos-besteffort 1000

Remember /proc from chapter 3: inside a container the main process is PID 1, so /proc/1/oom_score_adj belongs to the container's main program. It's a number from −1000 to 1000. When the machine runs out of memory, the kernel scores every process (mostly by how much memory it uses), adds this number, and kills the highest. Burstable values vary with the memory request (a bigger request gives a lower number).

#Eviction: the kubelet steps in first

The kernel's machine-wide OOM killer is the last resort. Before it gets there, the kubelet watches the node's free memory, roughly every 10 seconds:

memory.available = node capacity − working set of the whole node

If memory.available drops below evictionHard (100Mi by default), the kubelet marks the node with MemoryPressure and evicts a pod: stops it, marks it Failed, and doesn't restart it on this node.

To see it without waiting for a real shortage, the lab has a cluster profile that tells the kubelet to treat the node as low on memory once less than 75% of it is available. Nothing is edited on a live node: the setting is part of the cluster from the start. Then a BestEffort memory hog runs next to two small, well-behaved pods:

make clean && make cluster PROFILE=pressure
kubectl -n lab apply -f labs/18-node-pressure-eviction/pods.yaml
# wait a minute, then:
kubectl -n lab get pod memory-hog -o jsonpath='{.status.reason}: {.status.message}'

Evicted: The node was low on resource: memory. …
Container main was using 721996Ki, request is 0, has larger consumption of memory.

The two calm pods kept running. Roughly, the kubelet ranks pods by (1) whether their memory use is above their request, (2) their priority, and (3) how far above their request they are. The hog was using about 700 MiB against a request of 0, so it came first. Treat this as intuition; the Kubernetes docs have the full rules.

Lab: 18-node-pressure-eviction (needs make cluster PROFILE=pressure). Check what you built with make check LAB=18.

#In production

#19. Seeing memory in production

Question: which memory numbers should I watch, and which ones lie?

Like CPU in Part 2, every memory file has a Prometheus metric, exported by cAdvisor in the kubelet:

From chapter 17: container_memory_usage_bytes includes cache the kernel can drop, so it looks scary when nothing is wrong. Compare the working set with the limit; that's the number the kubelet uses for eviction, and the one that gets close to the limit before an OOM kill.

#Memory pressure

Part 2 introduced pressure: not how busy something is, but how long things waited. For memory, waiting means stalling while the kernel reclaims memory to make room. The lab's psi-memory pod rereads a 192 MiB file inside a 64Mi limit, so the kernel keeps throwing its cache away and reading it back:

kubectl -n lab exec psi-memory -- cat /sys/fs/cgroup/memory.pressure

some avg10=6.35 avg60=2.42 avg300=0.59 total=1967529
full avg10=6.35 avg60=2.42 avg300=0.59 total=1967529

some and full are the same because the pod has one busy process: whenever it waited, everything in the pod waited. About 6% of the last 10 seconds was lost to reclaim, and it hasn't been OOM-killed, because file cache can always be dropped. Memory that a program allocates can't, so for a leaking process, rising memory.pressure is often the earliest warning you'll get before an OOM kill.

Lab: 19-metrics-and-psi. Check what you built with make check LAB=19.

#Beyond CPU and memory

#20. The other limits, briefly

Question: is it only CPU and memory?

CPU and memory get all the attention, but a few more limits are worth knowing:

The PID limit matters more than it looks: every process and thread needs a PID, and a node only has so many. One runaway pod spawning processes (a fork bomb) can use them all, and then nothing on the node can start, not even the kubelet. And the last row is a common surprise: there is no cgroup for disk space. The kubelet adds up what each pod has written (logs, its writable layer and emptyDir volumes) and evicts it when it goes over.

Lab: 20-pids-and-disk (needs make cluster PROFILE=pids). Check what you built with make check LAB=20.

#21. Finale: resize a running pod

Question: if limits are just numbers in files, can we change them on a live pod?

Here's the proof that ties the whole series together. Since Kubernetes 1.33, where in-place resize became beta and on by default (it's been stable since 1.35), you can change a running pod's CPU and memory in place, with no restart. Remember chapter 8, where you changed cpu.max with echo while a process was running? That's all this is.

Try it yourself.

kubectl -n lab apply -f labs/21-in-place-resize/pod.yaml       # 200m CPU, 64Mi memory
kubectl -n lab wait --for=condition=Ready pod/resize-demo
kubectl -n lab exec resize-demo -- cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/memory.max

kubectl -n lab patch pod resize-demo --subresource resize --patch \
  '{"spec":{"containers":[{"name":"main","resources":{"requests":{"cpu":"500m","memory":"128Mi"},"limits":{"cpu":"500m","memory":"128Mi"}}}]}}'

kubectl -n lab exec resize-demo -- cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/memory.max
kubectl -n lab get pod resize-demo -o jsonpath='restarts={.status.containerStatuses[0].restartCount}{"\n"}'

20000 100000        ← before: 200m
67108864            ← before: 64 MiB
50000 100000        ← after: 500m
134217728           ← after: 128 MiB
restarts=0

The kubelet asked the runtime to rewrite the files of the live container. Same container, same process, new numbers. This was my favourite moment building the lab, because it's the clearest proof that resources: really is just a few numbers in a few files.

Lab: 21-in-place-resize. Check what you built with make check LAB=21.

#Wrap-up

#Put it to the test

Reading about throttling and knowing it when you see it are different skills. The lab has three small incidents that play like a short on-call shift. Each one sets the scene, shows you what the dashboard shows, and hands you a ticket that sounds like a real page, with a few places to look. Then it leaves the digging to you. It pauses between steps, so take your time.

| # | The ticket | Where the answer is | |:-:|---|---| | 1 | "Checkout is slow, but the CPU graph is under its limit" | Chapters 14 and 15 | | 2 | "OOMKilled, but the dashboard never went above half" | Chapters 16 and 17 | | 3 | "Pending for ages, the node's memory is almost empty" | Chapter 9 and 18 |

make incident-1                       # the scene, what the dashboard shows, then the ticket
make hint-1                           # stuck? one clue at a time; run it again for the next
make answer-1 CAUSE=<your answer>     # the evidence, why it happened, and the fix, applied live

Try to solve each one with only kubectl, cat and the files from this series before reaching for a clue.

#Make your app cgroup-aware

The kernel enforces the box, but the program inside should know how big the box is:

• Java: container support is on by default in JDK 10+ and 8u191+. Size the heap with -XX:MaxRAMPercentage rather than a fixed -Xmx.
• Go: from Go 1.25, GOMAXPROCS follows the container's CPU limit (rounded up, so 2.5 CPUs gives 3) and is re-checked if the limit changes. It ignores the CPU request, so with no limit it still uses every host core. On older versions, use go.uber.org/automaxprocs, or set GOMAXPROCS yourself.
• Node.js: set --max-old-space-size below the memory limit, leaving room for buffers.

#Cheat sheet

Three Kubernetes features often mentioned alongside these don't touch cgroups at all, so I've left them out of this series: ResourceQuota (a namespace's total budget), LimitRange (default and maximum requests per container) and PriorityClass (who gets scheduled, and who's evicted, first). They're checked by the API server and the scheduler (chapter 9). The lab has a hands-on extra for each: labs/E1-resourcequota, labs/E2-limitrange and labs/E3-priority-and-preemption.

#Takeaways

• Kubernetes decides; the kernel enforces. When something behaves strangely, ask which checkpoint did it, then read that layer's files.
• Requests place your pod, share CPU when it's contended, and decide eviction order. Make them honest.
• CPU limits throttle, even on an idle node, and all your threads spend the same budget. Throttling can trigger liveness restarts and readiness drop-outs that spread load to other pods. Many teams drop CPU limits for latency-sensitive services and rely on good requests instead. Measure throttling before and after.
• Memory limits end in OOM kills once reclaim runs out. Setting them is a good default, because one leaky container otherwise puts the whole node at risk. For important workloads, memory requests equal to limits give the most predictable behaviour, at the cost of reserving memory you may not always use.
• Watch the right numbers: the throttling ratio, the working set, oom_kill counts, and pressure. Average CPU and memory.current will mislead you.

If you made it through all three parts, thanks for reading. Clone the lab, work through the labs, and watch it all happen. And if something doesn't behave the way I described on your machine, open an issue or ping me on LinkedIn; I'd genuinely like to know.

#Glossary

#Lab command reference

Each chapter's lab lives in labs/, numbered like the chapters. Nothing is deleted behind your back: pods stay until you delete them or run make reset.

#Sources and further reading

• Linux kernel docs: Control Group v2, the reference for every file in this series
• Linux kernel docs: CFS bandwidth control, how quota, period, slices and burst work
• Linux kernel docs: EEVDF scheduler
• Linux kernel docs: Pressure stall information
• Kubernetes docs: Resource management for pods and containers
• Kubernetes docs: Node-pressure eviction
• Kubernetes docs: Resize CPU and memory resources assigned to containers
• Kubernetes blog: New conversion from cgroup v1 CPU shares to v2 CPU weight
• iximiuz Labs: Controlling process resources with Linux control groups, a great companion to chapter 8