Cgroups: A Deep Dive into Resource Management in Kubernetes – Part 1: Foundations

On a day when nothing had shipped and traffic was flat, one of our backend services quietly started restarting itself. A few times a day, at no time we could predict, its health probes would time out. Failed liveness probes got the pods restarted, and failed readiness probes pulled them out of rotation. The services calling it saw timeouts and p99 latency spikes.

We went through the usual suspects, and every one came back clean:

• CPU was below 80% of the pod's request. Nowhere near saturated.
• Memory was flat. No OOM kills.
• Deploys: nothing had shipped around the time it started.
• Traffic and log volume looked the same as before.

Only one metric was out of place: container_cpu_cfs_throttled_periods_total was spiking. When the team computed the throttle ratio, some pods were being throttled in about 40% of their CPU scheduling periods, while their CPU charts showed plenty of headroom.

Raising the pod's CPU limit from 5 to 8 cores brought throttling to almost zero. The restarts stopped and tail latency dropped sharply. No code changed.

So how can a pod be throttled on CPU while barely using its CPU? The answer is in cgroups and the Linux scheduler, the machinery underneath every container.

That's what this three-part series is about. We'll start from zero, and by the end you'll be able to:

• find the exact files on a node that hold your pod's CPU and memory limits, and read them;
• explain how a pod using a fraction of its CPU can still be frozen for most of every 100 ms, and how that turns into restarts and lost capacity;
• explain why a pod can be OOM-killed while the dashboard says it had room to spare;
• set requests and limits deliberately, not out of habit;
• resize a running pod's CPU and memory without restarting it, and know exactly why that works.

#How to read this series

The series is in three parts, and each one builds on the last:

Every chapter starts with a question, explains it with a sketch, lets you try it, and ends with one line to remember.

#The machine

Before we touch Kubernetes, we need to understand the machine it runs on. Everything in this series, from CPU throttling to OOM kills to that outage, comes from a few simple ideas about how Linux works. If you already know them, skim ahead. If you don't, this is the part that makes the rest easy.

docker run --rm -it --cpus=0.5 --memory=128m ubuntu bash

#1. What happens when you run a program?

A program is just a file on disk, for example /usr/bin/python3. It doesn't do anything until you run it.

When you run it, Linux loads it into memory and starts it. That running copy is a process. Run the same program twice and you get two processes. Each one gets a number, its PID (process ID), so the system can tell them apart.

A process can do several things at once by splitting its work into threads. A web server might have one thread per request, a few for background jobs and one for garbage collection. Threads are what actually run on the CPU.

That leads to the most important fact about CPUs:

One CPU core can run only one thread at any moment.

A laptop with 8 cores can run 8 threads at the same instant. Yet your machine has thousands of threads, and they all seem to be running. That's because something switches between them very quickly, giving each a few milliseconds, then the next, then the next, thousands of times a second.

That "something" is the kernel.

#So what is the kernel?

The kernel is the core of the operating system: the one program that controls the hardware. Your programs never touch the CPU or memory directly. They ask the kernel:

• "I'd like to run." The kernel's scheduler decides when, and on which core.
• "I need 50 MB of memory." The kernel decides whether to give it.

Since every request goes through the kernel, the kernel is the one place where limits can be enforced. When a container is "limited to half a CPU", nobody is watching it from outside. The kernel's scheduler simply stops giving it turns once it has used its share.

Lab: 01-processes-and-threads. Check what you built with make check LAB=01.

#2. Everything in Linux is a file

This is the idea that makes the rest of the post possible.

On Linux, almost everything is shown to you as a file, even things that aren't files. You read it with cat and change it with echo, the same tools you'd use for a text file.

Real files are on disk:

cat /etc/hostname          # a normal text file

Devices are files too. Your disk is /dev/sda, your terminal is /dev/tty, and there's a special file that throws away anything you write to it:

echo "gone" > /dev/null    # writing to a "file" that's really a black hole

And the kernel itself is files. Linux has special folders that aren't stored on disk at all. When you read a file inside them, the kernel creates the answer at that moment. When you write to one, the kernel changes a live setting.

Here's the proof:

ls -l /proc/meminfo

-r--r--r-- 1 root root 0 Oct 30 11:06 /proc/meminfo

The size is 0 bytes. Now read it:

head -3 /proc/meminfo

MemTotal:        8126480 kB
MemFree:         3512900 kB
MemAvailable:    6465152 kB

A file with zero bytes just showed you the machine's memory. Nothing was stored. The kernel worked the numbers out when you asked. Run it again and they'll be slightly different.

These special folders are called virtual filesystems. Two of them matter for this series: /proc and /sys.

Lab: 02-everything-is-a-file. Check what you built with make check LAB=02.

#3. /proc: what every process is doing right now

/proc is the kernel's live report on processes. Every running process gets its own folder, named after its PID:

ls /proc

1   8   9   buddyinfo   bus   cgroups   cmdline   config.gz   ...

The numbers are processes. Look inside one. /proc/self is a shortcut that always means "the process reading this", so you don't need to know your own PID:

cat /proc/self/comm                  # its name
grep Threads /proc/self/status       # how many threads it has
grep VmRSS  /proc/self/status        # how much memory it's using

cat
Threads:        1
VmRSS:          1544 kB

Tools like ps, top and free don't have secret access. They read these files and format them nicely.

Try the one that matters most for this series:

cat /proc/self/cgroup

0::/

Every process belongs to a cgroup, and this file says which one. Inside our container it's simply /. We'll see why in chapter 7. For now, note that the kernel tracks it for every process.

Lab: 03-proc. Check what you built with make check LAB=03.

#4. /sys: the kernel's settings panel

If /proc is mainly about processes, /sys is about the kernel's own components and settings: CPUs, disks, network cards, and the systems that manage them.

Its files are kept very simple: usually one value per file. That makes them easy to read and easy to change.

ls /sys

block  bus  class  dev  devices  firmware  fs  kernel  module

Some examples:

ls /sys/class/net                         # your network interfaces
cat /sys/class/net/eth0/address           # one interface's MAC address
cat /sys/devices/system/cpu/online        # which CPUs are online: 0-9 on my machine

For this series we only need one folder in /sys: /sys/fs, where the kernel keeps settings related to filesystems. Inside it is the folder this whole series is about:

ls /sys/fs

9p  bpf  btrfs  cgroup  erofs  ext4  f2fs  fuse  nfs  pstore  virtiofs  xfs

#/sys/fs/cgroup: where resource limits live

/sys/fs/cgroup is the kernel's resource control panel. Every limit on CPU, memory or number of processes, for every container on the machine, is a file somewhere under this folder.

cgroup is short for control group: a group of processes that the kernel limits and keeps count of together. Each cgroup is a folder under /sys/fs/cgroup. The files in it are its limits and counters, and every process listed in it shares them. Folders can hold folders, so cgroups nest into a tree.

You can check what kind of filesystem it is:

mount | grep cgroup

cgroup on /sys/fs/cgroup type cgroup2 (ro,nosuid,nodev,noexec,relatime)

type cgroup2 means cgroups version 2, the current version and the only one this series uses. You'll still meet version 1 on older nodes and in older articles, so here's how the two differ:

The ideas are the same in both; mostly the file names changed. So when an older article mentions cpu.shares or cpu.cfs_quota_us, read them as cpu.weight and cpu.max.

Two folders, two jobs. It's easy to mix them up, so here they are side by side:

When you want to know what a process is doing, look in /proc. When you want to know what it's allowed to do, look in /sys/fs/cgroup.

Lab: 04-sys-and-sys-fs-cgroup. Check what you built with make check LAB=04.

#5. Your first look at cpu.max and cpu.weight

Now open the panel. Inside our container:

ls /sys/fs/cgroup

cgroup.procs   cpu.max   cpu.stat   cpu.weight
memory.current   memory.max   memory.events   pids.max   ...

Each file is either a dial (a setting you can change) or a gauge (a reading the kernel keeps updated). The names follow one pattern: controller.setting. Everything starting with cpu. is about CPU, and everything starting with memory. is about memory.

Let's read the dials. Remember the container was started with --cpus=0.5 --memory=128m:

cat /sys/fs/cgroup/cpu.max

50000 100000

cat /sys/fs/cgroup/memory.max

134217728

There are your Docker flags, written as kernel settings.

cpu.max = 50000 100000. Two numbers, both in microseconds (millionths of a second):

• 100000 µs = 100 ms. This is the period: the kernel counts CPU use in 100 ms windows.
• 50000 µs = 50 ms. This is the quota: how much CPU time the group may use in each window.

So: "at most 50 ms of CPU in every 100 ms", which is half a CPU. That's --cpus=0.5. If there were no limit, the file would read max 100000.

memory.max = 134217728. This is in bytes: 128 × 1024 × 1024 = 134,217,728. That's --memory=128m. The group may never hold more than this.

Kernel files always use raw units like these. Keep this table nearby; the last two rows are the units you'll write in Kubernetes YAML:

Now the other CPU file:

cat /sys/fs/cgroup/cpu.weight

100

cpu.weight isn't a limit. It's a share. When several groups want the CPU at the same time, the kernel divides it in proportion to their weights. A group with weight 200 gets about twice as much as one with weight 100. When nobody else wants the CPU, the weight does nothing. 100 is the default.

These two CPU files are the most important pair in the post:

Next to them are the gauges, files the kernel keeps updating so you can see what's happening:

cat /sys/fs/cgroup/memory.current     # memory in use right now, in bytes
cat /sys/fs/cgroup/cpu.stat           # CPU used, and how often the budget ran out

Finally, how does the kernel know which processes these limits apply to? There's a file for that too:

cat /sys/fs/cgroup/cgroup.procs

1
…

Those are the PIDs in this group: 1 is your shell, and the other is the cat you just ran (its number will differ on your machine). The limits apply to every process listed here, and every thread in them, all together. Keep that in mind, because it's the key to the outage.

Lab: 05-cpu-max-and-cpu-weight. Check what you built with make check LAB=05.

#6. Why CPU limits slow you down but memory limits kill you

The kernel can make CPU "wait" but kills for memory because the two resources behave differently.

CPU is time. If a group has used its budget, the kernel can tell it to wait. Its threads pause and then continue in the next window. Nothing is lost; it just takes longer. This waiting is called throttling.

Memory is space. Once a program holds memory, the kernel can't make it "wait" for less. The data is already there. If the group is full and asks for more, the kernel first tries to free memory that can safely be dropped (such as cached copies of files). If that isn't enough, the only option left is to kill a process to free its memory. This is an OOM kill (out of memory).

That's why:

• a CPU limit that's too low makes your app slow,
• a memory limit that's too low makes your app crash.

Almost everything in the rest of this series follows from this difference.

Lab: 06-cpu-waits-memory-kills. Check what you built with make check LAB=06.

#7. So what is a container?

Now we can answer this properly. The Linux kernel has no such thing as a "container". No container object exists anywhere in the kernel. A container is an ordinary process with a few kernel features wrapped around it:

None of these pieces is new. chroot, which gives a process a different /, dates from Unix in 1979. FreeBSD's jails put a fence around a group of processes in 2000. Linux got its first namespace, for mounts, in 2002, and cgroups in 2008, built at Google under the name "process containers". What Docker added in 2013 was packaging: an image format and one command that puts all five together.

That job, putting the pieces together and starting your process, belongs to the container runtime (containerd, CRI-O, or Docker's own stack). In Kubernetes, the kubelet tells the runtime what to create, including the numbers from your resources: block. Chapter 10 follows those numbers all the way to the files.

This series is about one of the five pieces: cgroups. Namespaces show up only where they change what you see, as in the next section.

#Why /sys/fs/cgroup looked so small

Earlier, cat /proc/self/cgroup printed 0::/, as if we were at the top of the tree, and ls /sys/fs/cgroup showed only one group's files. That's a namespace at work: the cgroup namespace.

On the host machine, our container's cgroup is a folder deep in the tree, something like:

/sys/fs/cgroup/system.slice/docker-3f9a…e1.scope/cpu.max

The cgroup namespace makes that folder look like the top from inside the container. So /sys/fs/cgroup/cpu.max inside the container and that long path on the host are the same file. The namespace only changes the view. The limit is the same.

Lab: 07-what-is-a-container. Check what you built with make check LAB=07.

#Where we are

You now know enough to follow everything else:

• Programs run as threads, and the kernel hands out CPU and memory.
• Linux shows the kernel's settings as files: /proc for processes, /sys for settings.
• /sys/fs/cgroup holds the limits: cpu.max, cpu.weight, memory.max, and cgroup.procs for who they apply to.
• CPU limits cause waiting. Memory limits cause kills.
• A container is processes inside namespaces and cgroups.

So far we've only read these files. In the next section we write them.

#Hands on

#Set up the lab (5 minutes)

You need Docker Desktop (give it at least 4 CPUs and 6 GB of memory), plus kind and kubectl.

brew install kind kubectl          # on Windows, use WSL2: see the repo README
git clone https://github.com/NITHIN-JOHN-GEORGE/kubernetes-resources-under-the-hood
cd kubernetes-resources-under-the-hood
make check-setup                   # is everything ready? every ✗ comes with its fix
make cluster                       # a 2-node cluster inside Docker, plus the lab's own small image

The labs for chapters 1 to 7 need only Docker, so you can start those before creating the cluster. make cluster also builds lab-tools, a small image with readable commands the labs use: burn --threads 4 for busy loops, alloc for memory that's never freed, cache for file cache, and serve and load, a tiny web service and a load generator.

On Windows, run the same commands inside an Ubuntu terminal on WSL2. The repo README has the one extra setting WSL2 needs for cgroups v2.

What you get. kind ("Kubernetes in Docker") runs each Kubernetes node as a Docker container. make cluster creates a cluster called under-the-hood with two nodes:

• under-the-hood-control-plane runs the API server, the scheduler and the other control-plane parts.
• under-the-hood-worker is where every lab pod runs.

Because a kind node is just a Docker container, it has its own real cgroup tree and a real kubelet, the same pieces you'd find on a GKE or EKS node.

Two ways to look inside. You'll use these two commands all the time:

• kubectl exec <pod> -- <command> runs a command inside a container. It sees only its own cgroup (chapter 7), so kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.max shows exactly that container's CPU limit.
• docker exec under-the-hood-worker <command> runs a command on the node. From there you see the whole tree, including the folders for pods and QoS classes. On a real cluster this is the same as SSH-ing into the node.

Every chapter has a Try it yourself section with the exact commands and my output, and a Lab link to the matching folder in the repo, numbered like the chapters. Each lab's README walks through the same steps, and make check LAB=<number> tells you whether what you built behaves as described. (Chapters 1 to 7 are labs 01 to 07, which need only Docker.)

#8. Build a cgroup by hand

Question: if cgroups are just files, can we make one ourselves?

Yes. If you can make a cgroup with mkdir and echo, you'll recognise everything Kubernetes does later.

Think of a cgroup as a box with dials on the side: how much CPU, how much memory, how many processes. You put processes in the box, and the kernel keeps them within the dial settings.

Try it yourself. We'll use the worker node as a plain Linux machine, with no pods involved. Four steps are the whole idea:

Step 1: make the box.

docker exec under-the-hood-worker mkdir /sys/fs/cgroup/lab-cpu
docker exec under-the-hood-worker ls /sys/fs/cgroup/lab-cpu

cgroup.controllers
cgroup.events
…
cpu.max
cpu.stat
cpu.weight
…
memory.max
…
pids.max
…

A folder already full of files, 85 of them on my machine (trimmed above). You never create these yourself; the kernel adds them the moment the folder exists.

Step 2: set a dial. Allow 20 ms of CPU in every 100 ms, a fifth of one CPU:

docker exec under-the-hood-worker sh -c 'echo "20000 100000" > /sys/fs/cgroup/lab-cpu/cpu.max'

Step 3: put a process in. Start a shell that moves itself into the box ($$ is the shell's own PID), then spins forever:

docker exec -d under-the-hood-worker sh -c \
  'echo $$ > /sys/fs/cgroup/lab-cpu/cgroup.procs; while :; do :; done'
docker exec under-the-hood-worker cat /sys/fs/cgroup/lab-cpu/cgroup.procs

1074

That's the shell's PID on the node (yours will differ), now inside the box.

Writing a PID into cgroup.procs is how a process joins a cgroup. There's no other way, and it's exactly what a container runtime does for every container it starts.

Step 4: watch the kernel enforce it. cpu.stat keeps running totals. Read it, wait five seconds, read it again:

docker exec under-the-hood-worker grep -E 'usage_usec|nr_throttled' /sys/fs/cgroup/lab-cpu/cpu.stat
sleep 5
docker exec under-the-hood-worker grep -E 'usage_usec|nr_throttled' /sys/fs/cgroup/lab-cpu/cpu.stat

usage_usec 240916
nr_throttled 11
usage_usec 1243692
nr_throttled 62

usage_usec (CPU time used, in microseconds) went up by 1,002,776: one second of CPU in five seconds, so 20%. nr_throttled went up by 51: the loop hit its limit in every 100 ms window. It wants a whole CPU; the kernel gives it a fifth.

Now change the dial while it runs:

docker exec under-the-hood-worker sh -c 'echo "50000 100000" > /sys/fs/cgroup/lab-cpu/cpu.max'
sleep 5
docker exec under-the-hood-worker grep usage_usec /sys/fs/cgroup/lab-cpu/cpu.stat

usage_usec 3810766

That's 2,567,074 more in about five seconds: half a CPU. No restart, no API call. The kernel just reads the new number.

Hold on to that. In chapter 21 you'll do the same thing to a running Kubernetes pod with one kubectl command, and its container won't restart either, for exactly this reason.

And memory? Make a second box with a 50 MiB limit, and run a shell that keeps doubling a string, so its memory doubles every step:

docker exec under-the-hood-worker sh -c 'mkdir /sys/fs/cgroup/lab-mem && echo 50M > /sys/fs/cgroup/lab-mem/memory.max'
docker exec under-the-hood-worker sh -c 'echo 0 > /sys/fs/cgroup/lab-mem/memory.swap.max' 2>/dev/null
docker exec under-the-hood-worker sh -c 'echo $$ > /sys/fs/cgroup/lab-mem/cgroup.procs; a=x; while :; do a=$a$a; done'
echo "exit code: $?"
docker exec under-the-hood-worker cat /sys/fs/cgroup/lab-mem/memory.events

exit code: 137
low 0
high 0
max 35
oom 1
oom_kill 1
oom_group_kill 0

(The memory.swap.max line stops the kernel from dodging the limit by moving memory to disk.) In about a second the shell is gone, with exit code 137. memory.events tells the story: the limit was hit 35 times (max), reclaim couldn't free enough once (oom), and the kernel killed the process (oom_kill). That's an OOM kill, with no Kubernetes anywhere.

Clean up with kill and rmdir (a cgroup is removed like an empty folder):

docker exec under-the-hood-worker sh -c 'kill -9 $(cat /sys/fs/cgroup/lab-cpu/cgroup.procs); sleep 1; rmdir /sys/fs/cgroup/lab-cpu /sys/fs/cgroup/lab-mem'

Lab: 08-build-a-cgroup-by-hand. Check what you built with make check LAB=08.

#From YAML to kernel

In chapter 8 you made a cgroup with mkdir, set its limits with echo, and moved a process in by writing its PID to cgroup.procs. Kubernetes does exactly that for every container on every node, and nobody types a command. Your YAML just has to get past four checkpoints first.

Kubernetes never touches your running process. It never slows your app down and never kills it. It decides and translates: it takes your YAML, passes your numbers down a short chain of programs, and writes them into files on the node, the same files you just wrote by hand. Everything after that, the throttling and the OOM kills, is the Linux kernel reading those files.

#9. The journey of your YAML

Question: what happens to your resources: block between kubectl apply and a running process?

Between kubectl apply and your app running, your pod passes four checkpoints. Each one reads a different part of your YAML, makes one decision and, when the answer is no, fails in its own recognisable way:

Notice the split. Checkpoints 1 to 3 are deciders: they judge your pod against the state of the cluster and decide whether it's created, where it runs, and whether it gets to stay. None of them limits your app while it runs. Checkpoint 4 is the only enforcer: the kernel decides nothing about your pod, but it applies the numbers in its files to every slice of CPU time and every memory allocation, instantly.

Let's watch each one say no.

#Checkpoint 1, the API server: is this allowed here?

Before the pod is even saved, the API server checks it against the namespace's rules. Here the namespace has a quota of 100m CPU for requests, and the pod asks for 250m:

kubectl apply -f labs/09-four-checkpoints/quota.yaml
kubectl apply -f labs/09-four-checkpoints/pod-over-quota.yaml

Error from server (Forbidden): error when creating "labs/09-four-checkpoints/pod-over-quota.yaml": pods "over-quota" is forbidden: exceeded quota: tiny, requested: requests.cpu=250m, used: requests.cpu=0, limited: requests.cpu=100m

Turned away at the door. The pod was never created, so there's nothing to describe: this error is all you get.

#Checkpoint 2, the scheduler: which node has room?

The pod is saved but has no node yet. The scheduler looks for a node whose allocatable resources (what's left for pods after the system takes its share; chapter 18) can fit the pod's requests. It never looks at real usage. This pod requests 1000 CPUs:

kubectl -n lab apply -f labs/09-four-checkpoints/pod-too-big.yaml
kubectl -n lab get pod too-big
kubectl -n lab get events --field-selector involvedObject.name=too-big -o custom-columns=REASON:.reason,MESSAGE:.message

NAME      READY   STATUS    RESTARTS   AGE
too-big   0/1     Pending   0          3s
REASON             MESSAGE
FailedScheduling   0/2 nodes are available: 1 Insufficient cpu, 1 node(s) had untolerated taint(s). preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.

This time the pod exists, but it waits, and its events say why. (The second node is the control plane, which doesn't take ordinary pods.)

#Checkpoint 3, the kubelet: build the boxes, and keep the node healthy

Once a node is chosen, its kubelet creates the pod's cgroup and asks the container runtime to start each container with your numbers; chapter 10 follows them all the way down. Then the kubelet keeps watching the node. If memory runs short, it evicts pods to protect the rest, starting with those using the most beyond their memory request. Here's how that looks in chapter 18's lab, where a pod with no memory request kept growing:

Evicted: The node was low on resource: memory. Threshold quantity: 6094860Ki, available: 6090788Ki. Container main was using 1506200Ki, request is 0, has larger consumption of memory.

#Checkpoint 4, the kernel: over the limit, right now?

The first three checkpoints act once, or every few seconds. The kernel checks every slice of CPU time and every memory allocation, the whole time your app runs. A container that wants more CPU than its limit is throttled: made to wait. One that needs more memory than its limit is OOM-killed. Here's the pod's last state in chapter 16's lab:

OOMKilled exit=137

Kubernetes only reports that. The kernel did it.

#Which checkpoint did this?

Notice that the four checkpoints don't read the same things. Requests are read early, to place the pod and later to pick who to evict. Limits pass straight through to the kernel:

So when a pod misbehaves, the most useful first question is "which checkpoint did this?", and the symptom usually tells you:

Lab: 09-four-checkpoints. Check what you built with make check LAB=09.

#10. Follow one pod down to the kernel

Question: where exactly do 250m, 500m and 128Mi end up?

We'll use one pod for the rest of this section:

resources:
  requests:
    cpu: 250m
    memory: 64Mi
  limits:
    cpu: 500m
    memory: 128Mi

#Who writes the files

Kubernetes never writes your container's cgroup files itself. Your numbers are passed down a short chain, like a relay race, and each runner does one job before handing over:

1. The kubelet takes the resources: block from your pod and hands it to containerd. They talk over the CRI (Container Runtime Interface), the standard way Kubernetes talks to any container runtime.
2. containerd writes those numbers, with everything else about the container, into one JSON file, config.json, under linux.resources. Its format is set by the Open Container Initiative, so it's called the OCI runtime spec, and any OCI runtime can read it.
3. runc reads config.json, asks systemd to create the container's cgroup, writes your numbers into its files (cpu.max, cpu.weight, memory.max), starts your app inside it, and exits. A small helper, containerd-shim, stays behind as your app's parent.

The kubelet does create one cgroup itself: the pod's folder, which holds all of the pod's containers (chapter 11).

config.json describes the whole container: the command to run, the image's root filesystem, the namespaces, the capabilities. For this series only one part matters, linux.resources. Here's what it holds for our pod, and where runc puts each number:

The whole trip, in one table:

That last row surprises almost everyone I show this to. The memory request never even reaches config.json. By default, Kubernetes uses it to pick a node and, later, to decide who to evict first when the node runs short (chapter 18), but it doesn't ask the kernel to protect that memory. Since Kubernetes 1.37 the MemoryQoS feature can change that, but only if you opt in; chapter 17 covers it.

Try it yourself. Now check every step on a real pod. Create it and grab its container ID:

kubectl create namespace lab
kubectl -n lab apply -f labs/10-follow-one-pod-down/pod.yaml
kubectl -n lab wait --for=condition=Ready pod/follow-me
CID=$(kubectl -n lab get pod follow-me -o jsonpath='{.status.containerStatuses[0].containerID}' | sed 's#containerd://##')

Step 1, the spec runc got. crictl talks to containerd directly and prints the spec it handed to runc:

docker exec under-the-hood-worker crictl inspect $CID \
  | grep -E '"cgroupsPath"|"shares"|"quota"|"period"|"limit"'

"cgroupsPath": "kubelet-kubepods-burstable-pod97ea3d5e_c357_….slice:cri-containerd:…",
"period": 100000,
"quota": 50000,
"shares": 256
"limit": 134217728,

The same numbers as the sketch. cgroupsPath says where the container's cgroup goes: inside its pod's folder. shares: 256 is the 250m request in an older unit called CPU shares (chapter 13 explains it).

Step 2, systemd. One detail the sketch skips: on nodes that use the systemd cgroup driver (most do, including this one), runc doesn't make the folder itself. It asks systemd to create it as a scope, systemd's name for a group of processes started by someone else:

docker exec under-the-hood-worker systemctl show cri-containerd-$CID.scope \
  -p CPUWeight -p CPUQuotaPerSecUSec -p MemoryMax

CPUWeight=35
CPUQuotaPerSecUSec=500ms
MemoryMax=134217728

Same numbers, systemd's words: "500 ms of CPU per second" is our 500m limit.

Step 3, the kernel's files.

CDIR="$(./scripts/pod-cgroup.sh follow-me)/cri-containerd-$CID.scope"
docker exec under-the-hood-worker cat $CDIR/cpu.weight $CDIR/cpu.max $CDIR/memory.max

35
50000 100000
134217728

Now read the same files from inside the container, plus one more, memory.min:

kubectl -n lab exec follow-me -- cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/memory.max /sys/fs/cgroup/cpu.weight /sys/fs/cgroup/memory.min

50000 100000
134217728
35
0

Every number from the table, in the same kind of files you wrote by hand in chapter 8.

One process, two points of view. Find the container's process on the node, then look at it from inside the pod:

HPID=$(docker exec under-the-hood-worker head -n1 $CDIR/cgroup.procs)
docker exec under-the-hood-worker cat /proc/$HPID/comm /proc/$HPID/cgroup
kubectl -n lab exec follow-me -- cat /proc/1/comm /proc/self/cgroup

sleep
0::/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/…/cri-containerd-….scope
sleep
0::/

It's the same sleep. On the node it has an ordinary PID and a long cgroup path. Inside the pod, the PID namespace makes it PID 1 and the cgroup namespace makes its cgroup look like / (chapter 7).

Lab: 10-follow-one-pod-down. Check what you built with make check LAB=10.

#11. Where every pod lives: the cgroup tree

Question: why is a container's cgroup buried so deep?

Containers don't land just anywhere. The kubelet builds a tree, and the runtime adds each container's box at the bottom: one box for all pods, a box per QoS class (a label Kubernetes gives each pod, Guaranteed, Burstable or BestEffort, based on its requests and limits; chapter 18), a box per pod, and a box per container.

Nesting matters because anything in a box is also subject to the limits on the boxes around it. A container can't go past its pod's limit, and all pods together can't go past the ceiling on the top box. (Not every box sets a limit; the QoS boxes usually don't.)

Try it yourself. List the box that holds every pod on the worker:

docker exec under-the-hood-worker ls /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/
docker exec under-the-hood-worker cat /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/memory.max

You'll see kubelet-kubepods-burstable.slice and kubelet-kubepods-besteffort.slice among the files, and a memory.max of 8321515520 on my machine: about 7.75 GiB, the most that all pods together may use. Chapter 18 explains where that number comes from.

On GKE, EKS and most other clusters, the tree is the same but starts at /sys/fs/cgroup/kubepods.slice, without the kubelet- prefix (that's a kind detail).

This tree is also why a container's cgroup path is so long. Here's the follow-me pod from chapter 10, with each folder on its own line:

/sys/fs/cgroup/
  kubelet.slice/                                      kind's root for Kubernetes (kind only)
    kubelet-kubepods.slice/                           all pods
      kubelet-kubepods-burstable.slice/               the QoS class
        kubelet-kubepods-burstable-pod<uid>.slice/    the pod
          cri-containerd-<id>.scope/                  the container
            cpu.max                                   50000 100000

From inside the container, all of that is hidden. The cgroup namespace (chapter 7) makes the container's own folder look like the top, so the very same file is simply /sys/fs/cgroup/cpu.max, with the same 50000 100000 inside.

Lab: 11-the-cgroup-tree, which also shows the QoS oom_score_adj values from chapter 18. Check what you built with make check LAB=11.

#What's next

You can now read the files that hold every number in a resources: block: cpu.max, cpu.weight and memory.max, written by the kubelet into a tree of cgroups, one box per pod. You've even built one yourself.

But we haven't answered the question from the start: how does a pod get throttled on CPU it isn't even using? That's Part 2: CPU: how the kernel shares a CPU, what requests and limits really do, and the outage, explained step by step.