Cgroups: A Deep Dive into Resource Management in Kubernetes

On a day when nothing had shipped and traffic was flat, one of our backend services quietly started restarting itself. A few times a day, at no time we could predict, its health probes would time out. Failed liveness probes got the pods restarted, and failed readiness probes pulled them out of rotation. The services calling it saw timeouts and p99 latency spikes.

We went through the usual suspects, and every one came back clean:

• CPU was below 80% of the pod's request. Nowhere near saturated.
• Memory was flat. No OOM kills.
• Deploys: nothing had shipped around the time it started.
• Traffic and log volume looked the same as before.

Only one metric was out of place: container_cpu_cfs_throttled_periods_total was spiking. When the team computed the throttle ratio, some pods were being throttled in about 40% of their CPU scheduling periods, while their CPU charts showed plenty of headroom.

Raising the pod's CPU limit from 5 to 8 cores brought throttling to almost zero. The restarts stopped and tail latency dropped sharply. No code changed.

So how can a pod be throttled on CPU while barely using its CPU? The answer is in cgroups and the Linux scheduler, the machinery underneath every container.

That's what this post is about. We'll start from zero, and by the end you'll be able to:

• find the exact files on a node that hold your pod's CPU and memory limits, and read them;
• explain how a pod using a fraction of its CPU can still be frozen for most of every 100 ms, and how that turns into restarts and lost capacity;
• explain why a pod can be OOM-killed while the dashboard says it had room to spare;
• set requests and limits deliberately, not out of habit.

#How to read this post

The post is in six parts, and each one builds on the last:

Every chapter starts with a question, explains it with a sketch, lets you try it, and ends with one line to remember. Boxes marked Going deeper hold the extra detail. Skip them on a first read; nothing later depends on them.

#Part 1: Foundations

Before we touch Kubernetes, we need to understand the machine it runs on. Everything in this post, from CPU throttling to OOM kills to that outage, comes from a few simple ideas about how Linux works. If you already know them, skim ahead. If you don't, this is the part that makes the rest easy.

docker run --rm -it --cpus=0.5 --memory=128m ubuntu bash

#1. What happens when you run a program?

A program is just a file on disk, for example /usr/bin/python3. It doesn't do anything until you run it.

When you run it, Linux loads it into memory and starts it. That running copy is a process. Run the same program twice and you get two processes. Each one gets a number, its PID (process ID), so the system can tell them apart.

A process can do several things at once by splitting its work into threads. A web server might have one thread per request, a few for background jobs and one for garbage collection. Threads are what actually run on the CPU.

That leads to the most important fact about CPUs:

One CPU core can run only one thread at any moment.

A laptop with 8 cores can run 8 threads at the same instant. Yet your machine has thousands of threads, and they all seem to be running. That's because something switches between them very quickly, giving each a few milliseconds, then the next, then the next, thousands of times a second.

That "something" is the kernel.

#So what is the kernel?

The kernel is the core of the operating system: the one program that controls the hardware. Your programs never touch the CPU or memory directly. They ask the kernel:

• "I'd like to run." The kernel's scheduler decides when, and on which core.
• "I need 50 MB of memory." The kernel decides whether to give it.

Since every request goes through the kernel, the kernel is the one place where limits can be enforced. When a container is "limited to half a CPU", nobody is watching it from outside. The kernel's scheduler simply stops giving it turns once it has used its share.

Lab: 01-processes-and-threads. Check what you built with make check LAB=01.

#2. Everything in Linux is a file

This is the idea that makes the rest of the post possible.

On Linux, almost everything is shown to you as a file, even things that aren't files. You read it with cat and change it with echo, the same tools you'd use for a text file.

Real files are on disk:

cat /etc/hostname          # a normal text file

Devices are files too. Your disk is /dev/sda, your terminal is /dev/tty, and there's a special file that throws away anything you write to it:

echo "gone" > /dev/null    # writing to a "file" that's really a black hole

And the kernel itself is files. Linux has special folders that aren't stored on disk at all. When you read a file inside them, the kernel creates the answer at that moment. When you write to one, the kernel changes a live setting.

Here's the proof:

ls -l /proc/meminfo

-r--r--r-- 1 root root 0 Oct 30 11:06 /proc/meminfo

The size is 0 bytes. Now read it:

head -3 /proc/meminfo

MemTotal:        8126480 kB
MemFree:         3512900 kB
MemAvailable:    6465152 kB

A file with zero bytes just showed you the machine's memory. Nothing was stored. The kernel worked the numbers out when you asked. Run it again and they'll be slightly different.

These special folders are called virtual filesystems. Two of them matter for this post: /proc and /sys.

Lab: 02-everything-is-a-file. Check what you built with make check LAB=02.

#3. /proc: what every process is doing right now

/proc is the kernel's live report on processes. Every running process gets its own folder, named after its PID:

ls /proc

1   8   9   buddyinfo   bus   cgroups   cmdline   config.gz   ...

The numbers are processes. Look inside one. /proc/self is a shortcut that always means "the process reading this", so you don't need to know your own PID:

cat /proc/self/comm                  # its name
grep Threads /proc/self/status       # how many threads it has
grep VmRSS  /proc/self/status        # how much memory it's using

cat
Threads:        1
VmRSS:          1544 kB

Tools like ps, top and free don't have secret access. They read these files and format them nicely.

Try the one that matters most for this post:

cat /proc/self/cgroup

0::/

Every process belongs to a cgroup, and this file says which one. Inside our container it's simply /. We'll see why in chapter 7. For now, note that the kernel tracks it for every process.

Lab: 03-proc. Check what you built with make check LAB=03.

#4. /sys: the kernel's settings panel

If /proc is mainly about processes, /sys is about the kernel's own components and settings: CPUs, disks, network cards, and the systems that manage them.

Its files are kept very simple: usually one value per file. That makes them easy to read and easy to change.

ls /sys

block  bus  class  dev  devices  firmware  fs  kernel  module

Some examples:

ls /sys/class/net                         # your network interfaces
cat /sys/class/net/eth0/address           # one interface's MAC address
cat /sys/devices/system/cpu/online        # which CPUs are online: 0-9 on my machine

For this post we only need one folder in /sys: /sys/fs, where the kernel keeps settings related to filesystems. Inside it is the folder this whole post is about:

ls /sys/fs

9p  bpf  btrfs  cgroup  erofs  ext4  f2fs  fuse  nfs  pstore  virtiofs  xfs

#/sys/fs/cgroup: where resource limits live

/sys/fs/cgroup is the kernel's resource control panel. Every limit on CPU, memory or number of processes, for every container on the machine, is a file somewhere under this folder.

You can check what kind of filesystem it is:

mount | grep cgroup

cgroup on /sys/fs/cgroup type cgroup2 (ro,nosuid,nodev,noexec,relatime)

type cgroup2 means cgroups version 2, the current version and the only one this post uses. (Older systems used version 1, which spread the same settings across several folders. If you see tmpfs here instead, you're on v1.)

Lab: 04-sys-and-sys-fs-cgroup. Check what you built with make check LAB=04.

#5. Your first look at cpu.max and cpu.weight

Now open the panel. Inside our container:

ls /sys/fs/cgroup

cgroup.procs   cpu.max   cpu.stat   cpu.weight
memory.current   memory.max   memory.events   pids.max   ...

Each file is either a dial (a setting you can change) or a gauge (a reading the kernel keeps updated). The names follow one pattern: controller.setting. Everything starting with cpu. is about CPU, and everything starting with memory. is about memory.

Let's read the dials. Remember the container was started with --cpus=0.5 --memory=128m:

cat /sys/fs/cgroup/cpu.max

50000 100000

cat /sys/fs/cgroup/memory.max

134217728

There are your Docker flags, written as kernel settings.

cpu.max = 50000 100000. Two numbers, both in microseconds (millionths of a second):

• 100000 µs = 100 ms. This is the period: the kernel counts CPU use in 100 ms windows.
• 50000 µs = 50 ms. This is the quota: how much CPU time the group may use in each window.

So: "at most 50 ms of CPU in every 100 ms", which is half a CPU. That's --cpus=0.5. If there were no limit, the file would read max 100000.

memory.max = 134217728. This is in bytes: 128 × 1024 × 1024 = 134,217,728. That's --memory=128m. The group may never hold more than this.

Now the other CPU file:

cat /sys/fs/cgroup/cpu.weight

100

cpu.weight isn't a limit. It's a share. When several groups want the CPU at the same time, the kernel divides it in proportion to their weights. A group with weight 200 gets about twice as much as one with weight 100. When nobody else wants the CPU, the weight does nothing. 100 is the default.

These two CPU files are the most important pair in the post:

Next to them are the gauges, files the kernel keeps updating so you can see what's happening:

cat /sys/fs/cgroup/memory.current     # memory in use right now, in bytes
cat /sys/fs/cgroup/cpu.stat           # CPU used, and how often the budget ran out

Finally, how does the kernel know which processes these limits apply to? There's a file for that too:

cat /sys/fs/cgroup/cgroup.procs

1
…

Those are the PIDs in this group: 1 is your shell, and the other is the cat you just ran (its number will differ on your machine). The limits apply to every process listed here, and every thread in them, all together. Keep that in mind, because it's the key to the outage.

#What units do these files use?

Kernel files use raw units. Keep this table nearby:

Lab: 05-cpu-max-and-cpu-weight. Check what you built with make check LAB=05.

#6. Why CPU limits slow you down but memory limits kill you

The kernel can make CPU "wait" but kills for memory because the two resources behave differently.

CPU is time. If a group has used its budget, the kernel can tell it to wait. Its threads pause and then continue in the next window. Nothing is lost; it just takes longer. This waiting is called throttling.

Memory is space. Once a program holds memory, the kernel can't make it "wait" for less. The data is already there. If the group is full and asks for more, the kernel first tries to free memory that can safely be dropped (such as cached copies of files). If that isn't enough, the only option left is to kill a process to free its memory. This is an OOM kill (out of memory).

That's why:

• a CPU limit that's too low makes your app slow,
• a memory limit that's too low makes your app crash.

Almost everything in the rest of this post follows from this difference.

Lab: 06-cpu-waits-memory-kills. Check what you built with make check LAB=06.

#7. So what is a container?

Now we can answer this properly. The Linux kernel has no such thing as a "container". No container object exists anywhere in the kernel. A container is just ordinary processes that have been given two things:

A container runtime (Docker, containerd, CRI-O) sets both up and starts your process inside them. In Kubernetes, the kubelet tells the runtime what to create, including the numbers from your resources: block.

#Why /sys/fs/cgroup looked so small

Earlier, cat /proc/self/cgroup printed 0::/, as if we were at the top of the tree, and ls /sys/fs/cgroup showed only one group's files. That's a namespace at work: the cgroup namespace.

On the host machine, our container's cgroup is a folder deep in the tree, something like:

/sys/fs/cgroup/system.slice/docker-3f9a…e1.scope/cpu.max

The cgroup namespace makes that folder look like the top from inside the container. So /sys/fs/cgroup/cpu.max inside the container and that long path on the host are the same file. The namespace only changes the view. The limit is the same.

Lab: 07-what-is-a-container. Check what you built with make check LAB=07.

#Where we are

You now know enough to follow everything else:

• Programs run as threads, and the kernel hands out CPU and memory.
• Linux shows the kernel's settings as files: /proc for processes, /sys for settings.
• /sys/fs/cgroup holds the limits: cpu.max, cpu.weight, memory.max, and cgroup.procs for who they apply to.
• CPU limits cause waiting. Memory limits cause kills.
• A container is processes inside namespaces and cgroups.

So far we've only read these files. In Part 2 we write them.

#Part 2: cgroups by hand

#Set up the lab (5 minutes)

You need Docker Desktop (give it at least 4 CPUs and 6 GB of memory), plus kind and kubectl.

brew install kind kubectl          # on Windows, use WSL2: see the repo README
git clone https://github.com/NITHIN-JOHN-GEORGE/kubernetes-resources-under-the-hood
cd kubernetes-resources-under-the-hood
make check-setup                   # is everything ready? every ✗ comes with its fix
make cluster                       # a 2-node cluster inside Docker, plus the lab's own small image

The labs for chapters 1 to 7 need only Docker, so you can start those before creating the cluster. make cluster also builds lab-tools, a small image with readable commands the labs use: burn --threads 4 for busy loops, alloc for memory that's never freed, cache for file cache, and serve and load for chapter 15's outage.

On Windows, run the same commands inside an Ubuntu terminal on WSL2. The repo README has the one extra setting WSL2 needs for cgroups v2.

What you get. kind ("Kubernetes in Docker") runs each Kubernetes node as a Docker container. make cluster creates a cluster called under-the-hood with two nodes:

• under-the-hood-control-plane runs the API server, the scheduler and the other control-plane parts.
• under-the-hood-worker is where every lab pod runs.

Because a kind node is just a Docker container, it has its own real cgroup tree and a real kubelet, the same pieces you'd find on a GKE or EKS node.

Two ways to look inside. You'll use these two commands all the time:

• kubectl exec <pod> -- <command> runs a command inside a container. It sees only its own cgroup (chapter 7), so kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.max shows exactly that container's CPU limit.
• docker exec under-the-hood-worker <command> runs a command on the node. From there you see the whole tree, including the folders for pods and QoS classes. On a real cluster this is the same as SSH-ing into the node.

Every chapter has a Try it yourself section with the exact commands and my output, and a Lab link to the matching folder in the repo, numbered like the chapters. Each lab's README walks through the same steps, and make check LAB=<number> tells you whether what you built behaves as described. (Chapters 1 to 7 are labs 01 to 07, which need only Docker.)

#8. Build a cgroup by hand

Question: if cgroups are just files, can we make one ourselves?

Yes. If you can make a cgroup with mkdir and echo, you'll recognise everything Kubernetes does later.

Think of a cgroup as a box with dials on the side: how much CPU, how much memory, how many processes. You put processes in the box, and the kernel keeps them within the dial settings.

Try it yourself. We'll use the worker node as a plain Linux machine, with no pods involved. Four steps are the whole idea:

Step 1: make the box.

docker exec under-the-hood-worker mkdir /sys/fs/cgroup/lab-cpu
docker exec under-the-hood-worker ls /sys/fs/cgroup/lab-cpu

You get a folder already full of files: cpu.max, cpu.stat, memory.max, cgroup.procs and more. You never create these yourself; the kernel adds them the moment the folder exists.

Step 2: set a dial. Allow 20 ms of CPU in every 100 ms, a fifth of one CPU:

docker exec under-the-hood-worker sh -c 'echo "20000 100000" > /sys/fs/cgroup/lab-cpu/cpu.max'

Step 3: put a process in. Start a shell that moves itself into the box ($$ is the shell's own PID), then spins forever:

docker exec -d under-the-hood-worker sh -c \
  'echo $$ > /sys/fs/cgroup/lab-cpu/cgroup.procs; while :; do :; done'

Writing a PID into cgroup.procs is how a process joins a cgroup. There's no other way, and it's exactly what a container runtime does for every container it starts.

Step 4: watch the kernel enforce it. cpu.stat keeps running totals. Read it, wait five seconds, read it again:

docker exec under-the-hood-worker grep -E 'usage_usec|nr_throttled' /sys/fs/cgroup/lab-cpu/cpu.stat
sleep 5
docker exec under-the-hood-worker grep -E 'usage_usec|nr_throttled' /sys/fs/cgroup/lab-cpu/cpu.stat

usage_usec (CPU time used, in microseconds) goes up by about 1,000,000: one second of CPU in five seconds, so 20%. nr_throttled goes up by about 50: the loop hit its limit in every 100 ms window. On my machine: 203m of CPU and 51 throttled periods in 5 seconds. It wants a whole CPU; the kernel gives it a fifth.

Now change the dial while it runs:

docker exec under-the-hood-worker sh -c 'echo "50000 100000" > /sys/fs/cgroup/lab-cpu/cpu.max'

Read cpu.stat again and usage now climbs by about 2,500,000 every five seconds: half a CPU (I measured 512m). No restart, no API call. The kernel just reads the new number.

And memory? Make a second box with a 50 MiB limit, and run a shell that keeps doubling a string, so its memory doubles every step:

docker exec under-the-hood-worker sh -c 'mkdir /sys/fs/cgroup/lab-mem && echo 50M > /sys/fs/cgroup/lab-mem/memory.max'
docker exec under-the-hood-worker sh -c 'echo 0 > /sys/fs/cgroup/lab-mem/memory.swap.max' 2>/dev/null
docker exec under-the-hood-worker sh -c 'echo $$ > /sys/fs/cgroup/lab-mem/cgroup.procs; a=x; while :; do a=$a$a; done'
echo "exit code: $?"
docker exec under-the-hood-worker grep oom_kill /sys/fs/cgroup/lab-mem/memory.events

(The memory.swap.max line stops the kernel from dodging the limit by moving memory to disk.) In about a second, the shell is gone with exit code 137, and memory.events reads max 35, oom 1, oom_kill 1: the limit was hit 35 times, reclaim failed once, and the kernel killed the process. That's an OOM kill, with no Kubernetes anywhere.

Clean up with kill and rmdir (a cgroup is removed like an empty folder):

docker exec under-the-hood-worker sh -c 'kill -9 $(cat /sys/fs/cgroup/lab-cpu/cgroup.procs); sleep 1; rmdir /sys/fs/cgroup/lab-cpu /sys/fs/cgroup/lab-mem'

Lab: 08-build-a-cgroup-by-hand. Check what you built with make check LAB=08.

#Part 3: From YAML to kernel

#9. The four checkpoints

Question: what happens to your resources: block between kubectl apply and a running process?

Your YAML passes four checkpoints before it touches a process, and every chapter from here on sits on one of them.

1. API server. Before the pod is even saved, admission checks run. Does the namespace have a ResourceQuota, and would this pod go over it? Does a LimitRange need to fill in missing requests or limits? Which PriorityClass does it belong to? If any check fails, you get an error and the pod never exists.
2. Scheduler. The pod is saved but has no node yet. The scheduler looks for a node with enough allocatable resources (what's left for pods after the system takes its share; chapter 18) to fit the pod's requests. It doesn't look at real usage at all. If nothing fits, the pod stays Pending, or a lower-priority pod gets pushed out to make room.
3. Kubelet. The kubelet on the chosen node creates a cgroup for the pod and one for each container, writes your numbers into their files, and starts the containers inside them. After that it keeps watching the node: if memory or disk runs low, it evicts pods.
4. Kernel. From here on, every CPU turn and every memory allocation of your process is checked against those files. The kernel is what throttles CPU and OOM-kills processes.

When something goes wrong in production, asking "which checkpoint did this?" is usually the fastest way to the answer.

Lab: 09-four-checkpoints. Check what you built with make check LAB=09.

#10. Follow one pod down to the kernel

Question: where exactly do 250m, 500m and 128Mi end up?

We'll use one pod for the rest of Part 3:

resources:
  requests:
    cpu: 250m
    memory: 64Mi
  limits:
    cpu: 500m
    memory: 128Mi

Try it yourself. Create the pod and grab its container ID, which we'll use at every layer:

kubectl create namespace lab
kubectl -n lab apply -f labs/10-follow-one-pod-down/pod.yaml
kubectl -n lab wait --for=condition=Ready pod/follow-me
CID=$(kubectl -n lab get pod follow-me -o jsonpath='{.status.containerStatuses[0].containerID}' | sed 's#containerd://##')

Layer 1, the runtime: what the kubelet asked for. The kubelet doesn't create cgroups itself. It hands the container runtime (containerd) a description of the container in a standard format, the OCI runtime spec. crictl talks to the runtime directly:

docker exec under-the-hood-worker crictl inspect $CID \
  | grep -E '"cgroupsPath"|"shares"|"quota"|"period"|"limit"'

"cgroupsPath": "kubelet-kubepods-burstable-pod97ea3d5e_c357_….slice:cri-containerd:…",
"period": 100000,
"quota": 50000,
"shares": 256
"limit": 134217728,

That's your whole resources: block, translated. quota/period is the 500m CPU limit, limit is 128 MiB in bytes, and cgroupsPath says where the container's cgroup goes: inside its pod's folder. shares: 256 is the 250m request in an older unit called CPU shares (chapter 13 explains it).

Layer 2, systemd: the unit that owns the cgroup. On nodes that use the systemd cgroup driver (most do, including this one), the low-level runtime, runc, asks systemd to create the cgroup as a scope, systemd's name for a group of processes started by someone else:

docker exec under-the-hood-worker systemctl show cri-containerd-$CID.scope \
  -p CPUWeight -p CPUQuotaPerSecUSec -p MemoryMax

CPUWeight=35
CPUQuotaPerSecUSec=500ms
MemoryMax=134217728

Same numbers, systemd's words: "500 ms of CPU per second" is our 500m limit.

Layer 3, the kernel: the files.

CDIR="$(./scripts/pod-cgroup.sh follow-me)/cri-containerd-$CID.scope"
docker exec under-the-hood-worker cat $CDIR/cpu.weight $CDIR/cpu.max $CDIR/memory.max

35
50000 100000
134217728

Now read the same files from inside the container, plus one more, memory.min:

kubectl -n lab exec follow-me -- cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/memory.max /sys/fs/cgroup/cpu.weight /sys/fs/cgroup/memory.min

50000 100000
134217728
35
0

The same kind of files you wrote by hand in chapter 8. Here's the map to keep:

That last row surprises almost everyone I show this to. By default, Kubernetes uses the memory request to pick a node and, later, to decide who to evict first when the node runs short (chapter 18), but it doesn't ask the kernel to protect that memory. Since Kubernetes 1.37 the MemoryQoS feature can change that, but only if you opt in; chapter 17 covers it.

One process, two points of view. Find the container's process on the node, then look at it from inside the pod:

HPID=$(docker exec under-the-hood-worker head -n1 $CDIR/cgroup.procs)
docker exec under-the-hood-worker cat /proc/$HPID/comm /proc/$HPID/cgroup
kubectl -n lab exec follow-me -- cat /proc/1/comm /proc/self/cgroup

sleep
0::/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/…/cri-containerd-….scope
sleep
0::/

It's the same sleep. On the node it has an ordinary PID and a long cgroup path. Inside the pod, the PID namespace makes it PID 1 and the cgroup namespace makes its cgroup look like / (chapter 7).

Lab: 10-follow-one-pod-down. Check what you built with make check LAB=10.

#11. Where every pod lives: the cgroup tree

Question: why is a container's cgroup buried so deep?

The kubelet doesn't put containers just anywhere. It builds a tree: one box for all pods, a box per QoS class (a label Kubernetes gives each pod, Guaranteed, Burstable or BestEffort, based on its requests and limits; chapter 18), a box per pod, and a box per container.

Nesting matters because anything in a box is also subject to the limits on the boxes around it. A container can't go past its pod's limit, and all pods together can't go past the ceiling on the top box. (Not every box sets a limit; the QoS boxes usually don't.)

Try it yourself. List the box that holds every pod on the worker:

docker exec under-the-hood-worker ls /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/
docker exec under-the-hood-worker cat /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/memory.max

You'll see kubelet-kubepods-burstable.slice and kubelet-kubepods-besteffort.slice among the files, and a memory.max of 8321515520 on my machine: about 7.75 GiB, the most that all pods together may use. Chapter 18 explains where that number comes from.

On GKE, EKS and most other clusters, the tree is the same but starts at /sys/fs/cgroup/kubepods.slice, without the kubelet- prefix (that's a kind detail).

Lab: 11-the-cgroup-tree, which also shows the QoS oom_score_adj values from chapter 18. Check what you built with make check LAB=11.

#Part 4: CPU

#12. How Linux shares a CPU

Question: when more threads want to run than there are cores, who goes next?

This is the part most posts skip, and it's what makes CPU requests and limits make sense.

A core runs one thread at a time (chapter 1). Every thread that's ready to run waits in a queue, and the kernel's scheduler keeps picking the next one. The rule it uses is simple to state: keep a running tally of how much CPU each thread has had, and give the next turn to whoever has had the least.

That tally is called virtual runtime, and the word virtual is where weights come in. A thread with a higher weight has its tally grow more slowly. Run it for 10 ms with double the weight, and its tally only goes up by 5. So it keeps coming out as "had the least" more often, and ends up with roughly twice the CPU. Weight doesn't reserve anything; it just changes how fast your tally fills up.

cgroups plug straight into this. A cgroup is scheduled like one big thread with its own weight (cpu.weight), competing with its sibling cgroups. Whatever time the cgroup wins is then shared out among the threads inside it, the same way. That's how a pod's request turns into a share of the node's CPU.

There are two separate mechanisms, and they map neatly onto requests and limits:

Chapter 13 is about the first, chapter 14 about the second.

Lab: 12-how-linux-shares-a-cpu. Check what you built with make check LAB=12.

#13. CPU requests become weights

Question: what does a CPU request actually get you?

A request does not reserve a core. When the node is quiet it does nothing at all: anyone can use spare CPU. It only matters when everyone is hungry at once, and then the kernel cuts the pizza in proportion to each group's cpu.weight.

Try it yourself. Start two greedy pods with no CPU limits. Each runs one busy loop per CPU on the node, so together they want twice what exists. weight-low requests 100m, weight-high requests 800m:

kubectl -n lab apply -f labs/13-requests-become-weights/pods.yaml
kubectl -n lab wait --for=condition=Ready pod/weight-low pod/weight-high
kubectl -n lab exec weight-low  -- cat /sys/fs/cgroup/cpu.weight
kubectl -n lab exec weight-high -- cat /sys/fs/cgroup/cpu.weight

17
84

To see who actually wins, compare usage_usec for both pods over 10 seconds:

for p in weight-low weight-high; do kubectl -n lab exec $p -- grep usage_usec /sys/fs/cgroup/cpu.stat; done
sleep 10
for p in weight-low weight-high; do kubectl -n lab exec $p -- grep usage_usec /sys/fs/cgroup/cpu.stat; done

Subtract the first number from the second for each pod and divide by 10,000 to get millicores. On my 10-CPU setup the bigger request got 8.2 times more CPU: 8807m vs 1073m, the whole node between them. That's close to the 8× ratio of the requests, but not to 84 ÷ 17, which is only about 4.9. The next box explains why. Either way, treat it as a measurement, not a guarantee: the real split also depends on whatever else is competing on the node.

for p in weight-low weight-high; do
  echo "$p pod cgroup: $(docker exec under-the-hood-worker cat $(./scripts/pod-cgroup.sh $p)/cpu.weight)"
done

weight-low pod cgroup: 4
weight-high pod cgroup: 32

Lab: 13-requests-become-weights. Check what you built with make check LAB=13.

#14. CPU limits become a budget: throttling

Question: what happens when a container hits its CPU limit?

A limit is a different mechanism altogether: not a share, but a hard budget. Think of cpu.max as a wallet that's refilled every 100 milliseconds. With limits.cpu: 200m, the wallet gets 20 ms of CPU time per 100 ms period. Spend it early, and you wait until the next refill, even if the machine has nine idle CPUs.

Here's the trap, and it's the one I see most often: the wallet is shared by all threads in the cgroup. Four busy threads running on four CPUs spend 20 ms in about 5 ms of real time. After that, every thread in the group that's ready to run has to wait until the next period, about 95 ms later in this example. Your dashboard says "20% CPU", but every request that lands in one of those gaps is slow. That waiting is throttling.

#How the kernel hands out the budget

• At the start of each period, the cgroup's wallet is filled with its quota (20 ms here).
• Each CPU that's running one of the cgroup's threads takes a small slice out of the wallet (5 ms by default) and spends it, then comes back for more.
• When the wallet is empty, every CPU running that cgroup's threads stops running them, until a timer refills the wallet at the next period.

That's why the number of threads matters so much: four CPUs drain the wallet four times as fast as one.

#See it for yourself

Try it yourself. Two pods, both running four busy loops. cpu-limited has limits.cpu: 200m; cpu-unlimited has no limit:

kubectl -n lab apply -f labs/14-limits-and-throttling/pods.yaml
kubectl -n lab wait --for=condition=Ready pod/cpu-limited pod/cpu-unlimited
kubectl -n lab exec cpu-limited -- cat /sys/fs/cgroup/cpu.stat
sleep 10
kubectl -n lab exec cpu-limited -- cat /sys/fs/cgroup/cpu.stat

Between the two reads, nr_periods and nr_throttled both go up by about 100: throttled in almost every period. The unlimited pod's nr_throttled stays at 0. On my run, measured over 5 seconds on the node:

cpu-limited     throttled in 65 of 65 periods, 201m of CPU
cpu-unlimited   never throttled

To watch throttling as it happens, open a second terminal while the pods run:

./scripts/watch.sh cpu-limited

Every second it prints the limit from cpu.max, the CPU used and the throttled periods from cpu.stat, and memory and pressure, with the file each number came from.

Lab: 14-limits-and-throttling. Check what you built with make check LAB=14.

#15. The outage, explained

Question: how did a pod at "80% of request" end up restarting?

Back to the outage from the intro, one symptom at a time.

#Why the CPU graph looked healthy

Three things hid the problem:

• We compared usage with the wrong number. "Below 80% of request" measures against cpu.weight, which never throttles anything. Throttling only comes from cpu.max, the limit.
• Averages hide bursts. A one-minute CPU graph averages 600 periods. Throttling happens inside a single 100 ms period: a burst of threads empties the wallet in a few milliseconds, then everything waits. A pod can be quiet most of the time and still hit the wall whenever a burst arrives.
• "40% throttled" was 40% of busy periods (chapter 14), so it was even worse than it looked during traffic.

A quick worked example with request = 2 cores and limit = 5 cores. Suppose that in 25% of periods a burst spends the full 500 ms wallet, and in the rest the pod is fairly quiet at about 40 ms:

average = 0.25 × 5 cores + 0.75 × 0.4 cores = 1.55 cores
        → about 78% of a 2-core request: the dashboard says "fine"
        → but throttled in 25% of periods

#Why throttling broke the health probes

The kubelet runs three kinds of probes. All three are answered by threads inside the container, so all three freeze when the cgroup is throttled. What the kubelet does after a failure is different for each:

A probe is an HTTP request with a deadline (timeoutSeconds, 1 second by default). A single freeze of a few tens of milliseconds won't break that, but the effects stack up:

1. CPU time stretches into much more real time. If the container's threads can only run for the first 25 ms of each 100 ms period, a request that needs 100 ms of CPU takes about 400 ms to finish.
2. Queues fill up. Slow requests hold on to worker threads, and new requests, including the probe, wait behind them.
3. The probe misses its deadline failureThreshold times in a row (3 by default).

Liveness failure: the restart loop. The container is killed and restarted. Warming up (loading caches, opening connection pools, compiling code) is often the most parallel work a service ever does, so a fresh container throttles hardest of all.

Readiness failure: the quiet cascade. Nothing restarts, so it's harder to spot. The pod drops out of the Service, its traffic moves to the other pods, their bursts get bigger, they drain their wallets faster and start failing readiness too. Callers see timeouts and 503s.

#Why raising the limit fixed it

Going from 5 cores to 8 grew the wallet from 500 ms to 800 ms per period. Normal bursts now finished before it ran dry, so nr_throttled stopped climbing. We still saw a little throttling during warmup, which fits: warmup runs more threads in parallel than even 8 cores of budget covers.

#Why it started "out of nowhere"

Throttling depends on how many threads want to run at once, and that can change without anyone changing code:

• Bigger nodes. A container can see every core on the host. Modern runtimes (JDK 10+, Go 1.25+) size their thread pools from the CPU limit, but custom pools, native libraries and older runtimes often count host cores. Move to bigger nodes and the same service starts more threads against the same wallet.
• Burstier traffic at the same daily volume: synchronized retries, fan-out from callers, cron-driven clients.

We never confirmed the exact trigger, which is common, and it's why the throttling ratio belongs on a dashboard before you need it.

#What I check now

The throttle ratio, per pod:

sum by (pod) (rate(container_cpu_cfs_throttled_periods_total[5m]))
/
sum by (pod) (rate(container_cpu_cfs_periods_total[5m]))

How many seconds per second a pod spends frozen:

rate(container_cpu_cfs_throttled_seconds_total[5m])

And, when the image has ps, how many threads want a CPU right now (R means running or waiting to run):

kubectl exec <pod> -- sh -c 'for i in $(seq 20); do ps -eLo stat | grep -c ^R; sleep 0.2; done'

If that number regularly goes above your CPU limit in cores, the wallet is being drained. The lasting fix is to size thread pools and workers to the limit (or the request, if you drop the limit); raising the limit, like we did, buys time.

And for the probes:

• Keep liveness cheap and local ("is the process responsive?", not "are my dependencies up?"), and give it a more generous timeoutSeconds and failureThreshold than readiness, so a short throttling episode doesn't kill the pod.
• Use a startupProbe for slow warmups, so liveness can't kill a pod that's only slow because it's throttled while booting.
• Watch for readiness churn (pods flipping between ready and not ready) next to the throttle ratio. Churn with no restarts is the quiet version of this outage.

Try it yourself. Lab 15 recreates this outage on a laptop: three replicas of a small web service with a 500m CPU request and limit and liveness and readiness probes, hit by bursts of requests. On my run, a pod's average CPU came out at 384m against a 500m request: 77%, the same "below 80% of request" picture as our outage. Yet the kernel had throttled it in 790 of 843 periods. Within 90 seconds the slowest requests took over 4 seconds, and every pod had restarted two or three times. Raising the limit to 2 CPUs brought restarts to zero and the slowest request down to about half a second.

#Should you even set CPU limits?

This is one of the longest-running debates in Kubernetes, and after building this lab I understand why.

The case against. A CPU limit throttles you even when the node is idle, as the lab showed. And if your requests are honest, you're already protected when CPUs are busy: cpu.weight makes sure everyone gets their share. So a lot of teams set good CPU requests and simply drop CPU limits for latency-sensitive services. Spare CPU gets used instead of wasted, and those throttled gaps disappear.

The case for. Limits still earn their place when:

• you run untrusted or multi-tenant workloads and need hard isolation between them;
• you want predictable performance. Without a limit, a service is fast on a quiet node and slower on a busy one, which makes load tests misleading;
• you need the Guaranteed QoS class, for example for exclusive CPU cores with the static CPU manager, since Guaranteed requires limits equal to requests;
• it's a batch job you deliberately want capped.

What I do. Always set memory limits. For CPU, set honest requests everywhere, drop limits on latency-sensitive services, and keep them where isolation or predictability matters more than raw speed. Either way, watch the throttling ratio before and after the change. One catch: runtimes like Go and the JVM size their thread pools from the CPU limit, so without one they see every core on the host. Set GOMAXPROCS or -XX:ActiveProcessorCount explicitly to match your request.

If you want both sides argued in depth, learnk8s on CPU requests and limits and Why you should keep using CPU limits are good reads.

Lab: 15-the-outage. Check what you built with make check LAB=15.

#Part 5: Memory

CPU can make you wait. Memory can't (chapter 6), so it fails in a completely different way.

#16. Memory limits and the OOM killer

Question: what happens when a container reaches memory.max?

If a program needs more memory and its box is full, waiting doesn't help.

When a cgroup reaches memory.max, the kernel first tries to reclaim: it frees memory it can safely drop, such as cached file data. Only if that isn't enough does it call the OOM killer, which picks a process inside the cgroup and ends it.

Try it yourself. memory-hog has a 64Mi limit and runs a small Python script that grabs 10 MiB every 0.3 seconds and never lets go:

kubectl -n lab apply -f labs/16-memory-limits-and-oom/pod.yaml
kubectl -n lab get pod memory-hog -w

Within seconds the status flips to OOMKilled, Kubernetes restarts it, and it happens again. Press Ctrl-C and look at what it printed before it died:

kubectl -n lab logs memory-hog --previous | tail -3

allocated 40 MiB
allocated 50 MiB
allocated 60 MiB

It never reached 64. The Python interpreter itself lives in the same box and uses memory too.

Kubernetes records the death as reason: OOMKilled, exitCode: 137. Exit code 137 means "killed by a signal": 128 plus the signal number, and signal 9 is SIGKILL, the one a program can't catch or ignore. It gets no chance to clean up or write a last log line.

The kernel keeps its own tally in the pod's memory.events:

POD_DIR=$(./scripts/pod-cgroup.sh memory-hog)
docker exec under-the-hood-worker cat "$POD_DIR/memory.events"

low 0
high 0
max 75
oom 2
oom_kill 4
oom_group_kill 2

oom_group_kill shows up because, on cgroups v2, the kubelet sets memory.oom.group = 1 on each container by default (since Kubernetes 1.28): if one process has to die, the whole container goes, so nothing is left half-alive.

Lab: 16-memory-limits-and-oom. Check what you built with make check LAB=16.

#17. Not all memory is equal

Question: how can a pod be OOM-killed while the dashboard said it had room to spare?

This answers the third promise from the intro. The key is that a cgroup's memory comes in two kinds.

Both count towards memory.current and towards memory.max. But only one of them can get you killed.

Try it yourself. mem-lab has a 128Mi limit. First, read 192 MiB of files through it. The lab's cache command writes the file in 16 MiB steps, then reads it back twice:

kubectl -n lab apply -f labs/17-not-all-memory-is-equal/pod.yaml
kubectl -n lab wait --for=condition=Ready pod/mem-lab
kubectl -n lab exec mem-lab -- cache --mib 192
kubectl -n lab exec mem-lab -- cat /sys/fs/cgroup/memory.current
kubectl -n lab exec mem-lab -- grep -E '^(anon|file|inactive_file) ' /sys/fs/cgroup/memory.stat

On my run, memory.current read 125 MiB, almost exactly the limit, of which 122 MiB was file and under 1 MiB was anon. The pod's memory.events showed max 1814 (the limit was hit over and over) and oom_kill 0. Every time, the kernel dropped some old file cache and carried on. The container is "at its limit", and perfectly fine.

Now allocate memory that can't be dropped. The lab's alloc command grabs memory in steps and never lets go:

kubectl -n lab exec mem-lab -- alloc --mib 16 --every 100ms
kubectl -n lab get pod mem-lab

This time it ends with exit code 137 and a container restart, and memory.events moves to oom 1, oom_group_kill 1. There was no cache left to drop, only memory a running program was holding, so the only way to stay under the limit was to kill it.

#The working set: the number that matters

Because file cache inflates memory.current, the kubelet and most dashboards use the working set instead:

working set = memory.current − inactive_file

inactive_file is file cache that hasn't been touched recently, the first thing the kernel drops. The working set leaves it out, so it's a much better guess at what the container really needs. In the run above: memory.current 125 MiB, working set 1 MiB. Same container, same moment, two completely different stories. It's the number behind kubectl top and the metric container_memory_working_set_bytes.

#So why did the dashboard say there was room?

Usually one of three reasons, and now you can tell them apart:

1. A short spike. Most dashboards sample memory every 15 to 60 seconds, but the kernel reacts as soon as usage reaches memory.max. A spike between two samples never shows on the graph.
2. The wrong denominator. Many panels show memory as a share of the request or of the node, not of the limit.
3. The wrong kind of memory. A graph of memory.current mixes in reclaimable cache; a graph of the working set doesn't. They can tell very different stories.

memory.events doesn't miss anything, so when in doubt, trust it over the graph.

Lab: 17-not-all-memory-is-equal. Check what you built with make check LAB=17.

#18. When the whole node runs short

Question: who gets removed when a node runs low on memory?

The other way to run out is for the whole node to get low on memory, because many pods use more than they requested. Three mechanisms decide what happens then.

#Not all of a node is for pods

A node can't hand all its memory to pods; the operating system and Kubernetes' own programs need some too. What's left is called allocatable:

kubectl get node under-the-hood-worker \
  -o jsonpath='{.status.capacity.memory}{"  capacity\n"}{.status.allocatable.memory}{"  allocatable\n"}'

8126480Ki  capacity
8126480Ki  allocatable

They're equal on kind because it reserves nothing. On GKE or EKS, several hundred MiB or more are held back through three kubelet settings: systemReserved (for the OS), kubeReserved (for the kubelet and container runtime) and evictionHard (a safety margin). The scheduler only ever compares pod requests with allocatable, never real usage, and the kubelet writes capacity − systemReserved − kubeReserved into the memory.max of the box for all pods. That's the 8321515520 from chapter 11 (exactly 8,126,480 Ki).

#QoS classes: who is most protected

Kubernetes sorts every pod into one of three Quality of Service classes, worked out from its requests and limits:

The class shows up in two places. In the tree, each class gets its own box (chapter 11). And on every process, as a number the kernel reads when the whole machine is out of memory:

kubectl -n lab apply -f labs/11-the-cgroup-tree/pods.yaml
kubectl -n lab wait --for=condition=Ready pod/qos-guaranteed pod/qos-burstable pod/qos-besteffort
for p in qos-guaranteed qos-burstable qos-besteffort; do
  echo "$p $(kubectl -n lab exec $p -- cat /proc/1/oom_score_adj)"
done

qos-guaranteed -997
qos-burstable 996
qos-besteffort 1000

Remember /proc from chapter 3: inside a container the main process is PID 1, so /proc/1/oom_score_adj belongs to the container's main program. It's a number from −1000 to 1000. When the machine runs out of memory, the kernel scores every process (mostly by how much memory it uses), adds this number, and kills the highest. Burstable values vary with the memory request (a bigger request gives a lower number).

#Eviction: the kubelet steps in first

The kernel's machine-wide OOM killer is the last resort. Before it gets there, the kubelet watches the node's free memory, roughly every 10 seconds:

memory.available = node capacity − working set of the whole node

If memory.available drops below evictionHard (100Mi by default), the kubelet marks the node with MemoryPressure and evicts a pod: stops it, marks it Failed, and doesn't restart it on this node.

To see it without waiting for a real shortage, the lab has a cluster profile that tells the kubelet to treat the node as low on memory once less than 75% of it is available. Nothing is edited on a live node: the setting is part of the cluster from the start. Then a BestEffort memory hog runs next to two small, well-behaved pods:

make clean && make cluster PROFILE=pressure
kubectl -n lab apply -f labs/18-node-pressure-eviction/pods.yaml
# wait a minute, then:
kubectl -n lab get pod memory-hog -o jsonpath='{.status.reason}: {.status.message}'

Evicted: The node was low on resource: memory. …
Container main was using 721996Ki, request is 0, has larger consumption of memory.

The two calm pods kept running. Roughly, the kubelet ranks pods by (1) whether their memory use is above their request, (2) their priority, and (3) how far above their request they are. The hog was using about 700 MiB against a request of 0, so it came first. Treat this as intuition; the Kubernetes docs have the full rules.

Lab: 18-node-pressure-eviction (needs make cluster PROFILE=pressure). Check what you built with make check LAB=18.

#Part 6: In production

#19. Seeing it in production

Question: I can't cat files on production nodes all day. What do I watch instead?

The good news: every file in this post already has a metric. cAdvisor, built into the kubelet, reads the cgroup files and exports them to Prometheus:

#Pressure: how long things waited

There's one more family of files, and it answers a question usage can't. Usage tells you how busy something is. Pressure (PSI, short for pressure stall information) tells you how long things waited. Think of a shop with one till: "busy all day" sounds the same whether one customer was served or fifty queued out of the door. Pressure measures the queue.

Every cgroup has cpu.pressure, memory.pressure and io.pressure, and the node has the same in /proc/pressure/:

kubectl -n lab apply -f labs/19-metrics-and-psi/pods.yaml
kubectl -n lab wait --for=condition=Ready pod/psi-cpu pod/psi-memory
sleep 30
kubectl -n lab exec psi-cpu -- cat /sys/fs/cgroup/cpu.pressure

some avg10=59.96 avg60=24.36 avg300=6.05 total=18884381
full avg10=3.74 avg60=1.52 avg300=0.38 total=1359356

psi-cpu runs twice as many busy threads as the node has CPUs, so over the last 10 seconds at least one of its threads was waiting for a CPU 60% of the time. When a pod is slow but its CPU and memory graphs look fine, check pressure. Rising memory.pressure is often the earliest warning you'll get before an OOM kill. Since Kubernetes 1.36, the kubelet exports these numbers as stable metrics too, for nodes, pods and containers, through its Summary API and its /metrics/cadvisor endpoint.

Lab: 19-metrics-and-psi. Check what you built with make check LAB=19.

#20. The other limits, briefly

Question: is it only CPU and memory?

CPU and memory get all the attention, but a few more limits are worth knowing:

The PID limit matters more than it looks: every process and thread needs a PID, and a node only has so many. One runaway pod spawning processes (a fork bomb) can use them all, and then nothing on the node can start, not even the kubelet. And the last row is a common surprise: there is no cgroup for disk space. The kubelet adds up what each pod has written (logs, its writable layer and emptyDir volumes) and evicts it when it goes over.

Lab: 20-pids-and-disk (needs make cluster PROFILE=pids). Check what you built with make check LAB=20.

#21. Finale: resize a running pod

Question: if limits are just numbers in files, can we change them on a live pod?

Here's the proof that ties the whole post together. Since Kubernetes 1.33, where in-place resize became beta and on by default (it's been stable since 1.35), you can change a running pod's CPU and memory in place, with no restart. Remember chapter 8, where you changed cpu.max with echo while a process was running? That's all this is.

Try it yourself.

kubectl -n lab apply -f labs/21-in-place-resize/pod.yaml       # 200m CPU, 64Mi memory
kubectl -n lab wait --for=condition=Ready pod/resize-demo
kubectl -n lab exec resize-demo -- cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/memory.max

kubectl -n lab patch pod resize-demo --subresource resize --patch \
  '{"spec":{"containers":[{"name":"main","resources":{"requests":{"cpu":"500m","memory":"128Mi"},"limits":{"cpu":"500m","memory":"128Mi"}}}]}}'

kubectl -n lab exec resize-demo -- cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/memory.max
kubectl -n lab get pod resize-demo -o jsonpath='restarts={.status.containerStatuses[0].restartCount}{"\n"}'

20000 100000        ← before: 200m
67108864            ← before: 64 MiB
50000 100000        ← after: 500m
134217728           ← after: 128 MiB
restarts=0

The kubelet asked the runtime to rewrite the files of the live container. Same container, same process, new numbers. This was my favourite moment building the lab, because it's the clearest proof that resources: really is just a few numbers in a few files.

Lab: 21-in-place-resize. Check what you built with make check LAB=21.

#Wrap-up

#Put it to the test

Reading about throttling and knowing it when you see it are different skills. The lab has three small incidents that play like a short on-call shift. Each one sets the scene, shows you what the dashboard shows, and hands you a ticket that sounds like a real page, with a few places to look. Then it leaves the digging to you. It pauses between steps, so take your time.

| # | The ticket | Where the answer is | |:-:|---|---| | 1 | "Checkout is slow, but the CPU graph is under its limit" | Chapters 14 and 15 | | 2 | "OOMKilled, but the dashboard never went above half" | Chapters 16 and 17 | | 3 | "Pending for ages, the node's memory is almost empty" | Chapters 9 and 18 |

make incident-1                       # the scene, what the dashboard shows, then the ticket
make hint-1                           # stuck? one clue at a time; run it again for the next
make answer-1 CAUSE=<your answer>     # the evidence, why it happened, and the fix, applied live

Try to solve each one with only kubectl, cat and the files from this post before reaching for a clue.

#Make your app cgroup-aware

The kernel enforces the box, but the program inside should know how big the box is:

• Java: container support is on by default in JDK 10+ and 8u191+. Size the heap with -XX:MaxRAMPercentage rather than a fixed -Xmx.
• Go: from Go 1.25, GOMAXPROCS follows the container's CPU limit (rounded up, so 2.5 CPUs gives 3) and is re-checked if the limit changes. It ignores the CPU request, so with no limit it still uses every host core. On older versions, use go.uber.org/automaxprocs, or set GOMAXPROCS yourself.
• Node.js: set --max-old-space-size below the memory limit, leaving room for buffers.

#Cheat sheet

Three Kubernetes features often mentioned alongside these don't touch cgroups at all, so I've left them out of this post: ResourceQuota (a namespace's total budget), LimitRange (default and maximum requests per container) and PriorityClass (who gets scheduled, and who's evicted, first). They're checked by the API server and the scheduler (chapter 9). The lab has a hands-on extra for each: labs/E1-resourcequota, labs/E2-limitrange and labs/E3-priority-and-preemption.

#Takeaways

• Kubernetes decides; the kernel enforces. When something behaves strangely, ask which checkpoint did it, then read that layer's files.
• Requests place your pod, share CPU when it's contended, and decide eviction order. Make them honest.
• CPU limits throttle, even on an idle node, and all your threads spend the same budget. Throttling can trigger liveness restarts and readiness drop-outs that spread load to other pods. Many teams drop CPU limits for latency-sensitive services and rely on good requests instead. Measure throttling before and after.
• Memory limits end in OOM kills once reclaim runs out. Setting them is a good default, because one leaky container otherwise puts the whole node at risk. For important workloads, memory requests equal to limits give the most predictable behaviour, at the cost of reserving memory you may not always use.
• Watch the right numbers: the throttling ratio, the working set, oom_kill counts, and pressure. Average CPU and memory.current will mislead you.

If you made it this far, thanks for reading. Clone the lab, work through the labs, and watch it all happen. And if something doesn't behave the way I described on your machine, open an issue or ping me on LinkedIn; I'd genuinely like to know.

#Glossary

#Lab command reference

Each chapter's lab lives in labs/, numbered like the chapters. Nothing is deleted behind your back: pods stay until you delete them or run make reset.

#Sources and further reading

• Linux kernel docs: Control Group v2, the reference for every file in this post
• Linux kernel docs: CFS bandwidth control, how quota, period, slices and burst work
• Linux kernel docs: EEVDF scheduler
• Linux kernel docs: Pressure stall information
• Kubernetes docs: Resource management for pods and containers
• Kubernetes docs: Node-pressure eviction
• Kubernetes docs: Resize CPU and memory resources assigned to containers
• Kubernetes blog: New conversion from cgroup v1 CPU shares to v2 CPU weight
• iximiuz Labs: Controlling process resources with Linux control groups, a great companion to chapter 8