Cgroups: A Deep Dive into Resource Management in Kubernetes – Part 2: CPU

In Part 1 we built the foundations. In short:

• A cgroup is a folder under /sys/fs/cgroup, and its files are limits the kernel enforces.
• The kubelet turns requests.cpu into cpu.weight and limits.cpu into cpu.max, for every pod, in a tree of cgroups (one box per pod, one per container).
• CPU can be made to wait. Memory can't, so it ends in kills.

Part 1 opened with an outage: a service throttled in about 40% of its CPU periods while its CPU graph showed plenty of headroom, until raising its limit made the restarts stop. This part explains exactly how that happens.

In this part:

• Sharing a CPU: how the kernel divides CPU time, and how your request becomes a weight.
• Limits and throttling: how your limit becomes a budget, and the outage, explained step by step.
• In production: the metrics that show throttling before your users do.

To follow along, set up the lab from Part 1 first.

#Sharing a CPU

#12. How Linux shares a CPU

Question: when more threads want to run than there are cores, who goes next?

This is the part most posts skip, and it's what makes CPU requests and limits make sense.

A core runs one thread at a time (chapter 1). Every thread that's ready to run waits in a queue, and the kernel's scheduler keeps picking the next one. The rule it uses is simple to state: keep a running tally of how much CPU each thread has had, and give the next turn to whoever has had the least.

That tally is called virtual runtime, and the word virtual is where weights come in. A thread with a higher weight has its tally grow more slowly. Run it for 10 ms with double the weight, and its tally only goes up by 5. So it keeps coming out as "had the least" more often, and ends up with roughly twice the CPU. Weight doesn't reserve anything; it just changes how fast your tally fills up.

cgroups plug straight into this. A cgroup is scheduled like one big thread with its own weight (cpu.weight), competing with its sibling cgroups. Whatever time the cgroup wins is then shared out among the threads inside it, the same way. That's how a pod's request turns into a share of the node's CPU.

There are two separate mechanisms, and they map neatly onto requests and limits:

Chapter 13 is about the first, chapter 14 about the second.

Lab: 12-how-linux-shares-a-cpu. Check what you built with make check LAB=12.

#13. CPU requests become weights

Question: what does a CPU request actually get you?

A request does not reserve a core. When the node is quiet it does nothing at all: anyone can use spare CPU. It only matters when everyone is hungry at once, and then the kernel cuts the pizza in proportion to each group's cpu.weight.

Try it yourself. Start two greedy pods with no CPU limits. Each runs one busy loop per CPU on the node, so together they want twice what exists. weight-low requests 100m, weight-high requests 800m:

kubectl -n lab apply -f labs/13-requests-become-weights/pods.yaml
kubectl -n lab wait --for=condition=Ready pod/weight-low pod/weight-high
kubectl -n lab exec weight-low  -- cat /sys/fs/cgroup/cpu.weight
kubectl -n lab exec weight-high -- cat /sys/fs/cgroup/cpu.weight

17
84

To see who actually wins, compare usage_usec for both pods over 10 seconds:

for p in weight-low weight-high; do kubectl -n lab exec $p -- grep usage_usec /sys/fs/cgroup/cpu.stat; done
sleep 10
for p in weight-low weight-high; do kubectl -n lab exec $p -- grep usage_usec /sys/fs/cgroup/cpu.stat; done

Subtract the first number from the second for each pod and divide by 10,000 to get millicores. On my 10-CPU setup the bigger request got 8.2 times more CPU: 8807m vs 1073m, the whole node between them. That's close to the 8× ratio of the requests, but not to 84 ÷ 17, which is only about 4.9. The next box explains why. Either way, treat it as a measurement, not a guarantee: the real split also depends on whatever else is competing on the node.

weight = 1 + (shares − 2)
             × 9999 / 262142

for p in weight-low weight-high; do
  echo "$p pod cgroup: $(docker exec under-the-hood-worker cat $(./scripts/pod-cgroup.sh $p)/cpu.weight)"
done

weight-low pod cgroup: 4
weight-high pod cgroup: 32

Lab: 13-requests-become-weights. Check what you built with make check LAB=13.

#Limits and throttling

#14. CPU limits become a budget: throttling

Question: what happens when a container hits its CPU limit?

A limit is a different mechanism altogether: not a share, but a hard budget. Think of cpu.max as a wallet that's refilled every 100 milliseconds. With limits.cpu: 200m, the wallet gets 20 ms of CPU time per 100 ms period. Spend it early, and you wait until the next refill, even if the machine has nine idle CPUs.

Here's the trap, and it's the one I see most often: the wallet is shared by all threads in the cgroup. Four busy threads running on four CPUs spend 20 ms in about 5 ms of real time. After that, every thread in the group that's ready to run has to wait until the next period, about 95 ms later in this example. Your dashboard says "20% CPU", but every request that lands in one of those gaps is slow. That waiting is throttling.

#How the kernel hands out the budget

• At the start of each period, the cgroup's wallet is filled with its quota (20 ms here).
• Each CPU that's running one of the cgroup's threads takes a small slice out of the wallet (5 ms by default) and spends it, then comes back for more.
• When the wallet is empty, every CPU running that cgroup's threads stops running them, until a timer refills the wallet at the next period.

That's why the number of threads matters so much: four CPUs drain the wallet four times as fast as one.

This wallet has a name: the CFS bandwidth controller, the part of the Linux scheduler that enforces cpu.max. It was built for CFS, the scheduler from chapter 12. When EEVDF replaced CFS in Linux 6.6, the bandwidth controller and its files stayed exactly the same. That's why the throttling metrics you'll see in production still have cfs in their names, like container_cpu_cfs_throttled_periods_total.

#See it for yourself

Try it yourself. Two pods, both running four busy loops. cpu-limited has limits.cpu: 200m; cpu-unlimited has no limit:

kubectl -n lab apply -f labs/14-limits-and-throttling/pods.yaml
kubectl -n lab wait --for=condition=Ready pod/cpu-limited pod/cpu-unlimited
kubectl -n lab exec cpu-limited -- cat /sys/fs/cgroup/cpu.stat
sleep 10
kubectl -n lab exec cpu-limited -- cat /sys/fs/cgroup/cpu.stat

Between the two reads, nr_periods and nr_throttled both go up by about 100: throttled in almost every period. The unlimited pod's nr_throttled stays at 0. On my run, measured over 5 seconds on the node:

cpu-limited     throttled in 65 of 65 periods, 201m of CPU
cpu-unlimited   never throttled

To watch throttling as it happens, open a second terminal while the pods run:

./scripts/watch.sh cpu-limited

Every second it prints the limit from cpu.max, the CPU used and the throttled periods from cpu.stat, and memory and pressure, with the file each number came from.

Lab: 14-limits-and-throttling. Check what you built with make check LAB=14.

Back to the outage from Part 1's intro, one symptom at a time. How did a pod at "80% of request" end up restarting?

#Why the CPU graph looked healthy

Three things hid the problem:

• We compared usage with the wrong number. "Below 80% of request" measures against cpu.weight, which never throttles anything. Throttling only comes from cpu.max, the limit.
• Averages hide bursts. A one-minute CPU graph averages 600 periods. Throttling happens inside a single 100 ms period: a burst of threads empties the wallet in a few milliseconds, then everything waits. A pod can be quiet most of the time and still hit the wall whenever a burst arrives.
• "40% throttled" was 40% of busy periods (chapter 14), so it was even worse than it looked during traffic.

A quick worked example with request = 2 cores and limit = 5 cores. Suppose that in 25% of periods a burst spends the full 500 ms wallet, and in the rest the pod is fairly quiet at about 40 ms:

average = 0.25 × 5 cores + 0.75 × 0.4 cores = 1.55 cores
        → about 78% of a 2-core request: the dashboard says "fine"
        → but throttled in 25% of periods

#Why throttling broke the health probes

The kubelet runs three kinds of probes. All three are answered by threads inside the container, so all three freeze when the cgroup is throttled. What the kubelet does after a failure is different for each:

A probe is an HTTP request with a deadline (timeoutSeconds, 1 second by default). A single freeze of a few tens of milliseconds won't break that, but the effects stack up:

1. CPU time stretches into much more real time. If the container's threads can only run for the first 25 ms of each 100 ms period, a request that needs 100 ms of CPU takes about 400 ms to finish.
2. Queues fill up. Slow requests hold on to worker threads, and new requests, including the probe, wait behind them.
3. The probe misses its deadline failureThreshold times in a row (3 by default).

Liveness failure: the restart loop. The container is killed and restarted. Warming up (loading caches, opening connection pools, compiling code) is often the most parallel work a service ever does, so a fresh container throttles hardest of all.

Readiness failure: the quiet cascade. Nothing restarts, so it's harder to spot. The pod drops out of the Service, its traffic moves to the other pods, their bursts get bigger, they drain their wallets faster and start failing readiness too. Callers see timeouts and 503s.

#Why raising the limit fixed it

Going from 5 cores to 8 grew the wallet from 500 ms to 800 ms per period. Normal bursts now finished before it ran dry, so nr_throttled stopped climbing. We still saw a little throttling during warmup, which fits: warmup runs more threads in parallel than even 8 cores of budget covers.

#Why it started "out of nowhere"

Throttling depends on how many threads want to run at once, and that can change without anyone changing code:

• Bigger nodes. A container can see every core on the host. Modern runtimes (JDK 10+, Go 1.25+) size their thread pools from the CPU limit, but custom pools, native libraries and older runtimes often count host cores. Move to bigger nodes and the same service starts more threads against the same wallet.
• Burstier traffic at the same daily volume: synchronized retries, fan-out from callers, cron-driven clients.

We never confirmed the exact trigger, which is common, and it's why the throttling ratio belongs on a dashboard before you need it.

#What I check now

The throttle ratio, per pod:

sum by (pod) (rate(container_cpu_cfs_throttled_periods_total[5m]))
/
sum by (pod) (rate(container_cpu_cfs_periods_total[5m]))

How many seconds per second a pod spends frozen:

rate(container_cpu_cfs_throttled_seconds_total[5m])

And, when the image has ps, how many threads want a CPU right now (R means running or waiting to run):

kubectl exec <pod> -- sh -c 'for i in $(seq 20); do ps -eLo stat | grep -c ^R; sleep 0.2; done'

If that number regularly goes above your CPU limit in cores, the wallet is being drained. The lasting fix is to size thread pools and workers to the limit (or the request, if you drop the limit); raising the limit, like we did, buys time.

And for the probes:

• Keep liveness cheap and local ("is the process responsive?", not "are my dependencies up?"), and give it a more generous timeoutSeconds and failureThreshold than readiness, so a short throttling episode doesn't kill the pod.
• Use a startupProbe for slow warmups, so liveness can't kill a pod that's only slow because it's throttled while booting.
• Watch for readiness churn (pods flipping between ready and not ready) next to the throttle ratio. Churn with no restarts is the quiet version of this outage.

Try it yourself. Lab 15 recreates this outage on a laptop: three replicas of a small web service with a 500m CPU request and limit and liveness and readiness probes, hit by bursts of requests. On my run, a pod's average CPU came out at 384m against a 500m request: 77%, the same "below 80% of request" picture as our outage. Yet the kernel had throttled it in 790 of 843 periods. Within 90 seconds the slowest requests took over 4 seconds, and every pod had restarted two or three times. Raising the limit to 2 CPUs brought restarts to zero and the slowest request down to about half a second.

#Should you even set CPU limits?

My answer, after building this lab: for latency-sensitive services, usually no. Set honest CPU requests, drop the CPU limit, and watch the throttling ratio fall to zero. In lab 15, raising the limit to 2 CPUs took restarts from two or three per pod to none, and the slowest request from over 4 seconds to about half a second. With no limit at all, cpu.max reads max: there's no budget to run out of, so nothing is ever throttled.

You aren't giving up fairness. When CPUs are busy, cpu.weight still gives every pod its share in proportion to its request. What you give up is the forced idling: a limit throttles you even when the node has CPU to spare.

Many teams' manifests carry CPU limits because they were copied from somewhere else, not because anyone decided to throttle. Keep a CPU limit only when you can say why:

• you run untrusted or multi-tenant workloads and need hard isolation between them;
• you want predictable performance. Without a limit, a service is fast on a quiet node and slower on a busy one, which makes load tests misleading;
• you need the Guaranteed QoS class, for example for exclusive CPU cores with the static CPU manager, since Guaranteed requires limits equal to requests;
• it's a batch job you deliberately want capped.

Two things don't change. Always set memory limits: memory can't be shared out like CPU, and one leak can take down the node (Part 3). And if you drop the CPU limit, tell your runtime how many cores to use: Go and the JVM size their thread pools from the limit, so without one they see every core on the host. Set GOMAXPROCS or -XX:ActiveProcessorCount to match your request.

If you want both sides argued in depth, learnk8s on CPU requests and limits and Why you should keep using CPU limits are good reads.

Lab: 15-the-outage. Check what you built with make check LAB=15.

#In production

#Watching CPU in production

Question: I can't cat files on production nodes all day. What do I watch instead?

The good news: every CPU file in this part already has a metric. cAdvisor, built into the kubelet, reads the cgroup files and exports them to Prometheus:

The one to alert on is the throttle ratio from chapter 14, the share of periods in which the container ran out of budget:

rate(container_cpu_cfs_throttled_periods_total[5m])
  / rate(container_cpu_cfs_periods_total[5m])

Our outage would have shown about 0.4 here, while average CPU looked fine.

#Pressure: how long things waited

There's one more family of files, and it answers a question usage can't. Usage tells you how busy something is. Pressure (PSI, short for pressure stall information) tells you how long things waited. Think of a shop with one till: "busy all day" sounds the same whether one customer was served or fifty queued out of the door. Pressure measures the queue.

Every cgroup has cpu.pressure, memory.pressure and io.pressure, and the node has the same in /proc/pressure/. Here's CPU:

kubectl -n lab apply -f labs/19-metrics-and-psi/pods.yaml
kubectl -n lab wait --for=condition=Ready pod/psi-cpu pod/psi-memory
sleep 30
kubectl -n lab exec psi-cpu -- cat /sys/fs/cgroup/cpu.pressure

some avg10=59.96 avg60=24.36 avg300=6.05 total=18884381
full avg10=3.74 avg60=1.52 avg300=0.38 total=1359356

psi-cpu has no CPU limit, so it's never throttled. It runs twice as many busy threads as the node has CPUs, so over the last 10 seconds at least one of its threads was waiting for a CPU 60% of the time. That's waiting a usage graph can't show you. When a pod is slow but its CPU graph looks fine, check cpu.pressure. Since Kubernetes 1.36, the kubelet exports these numbers as stable metrics too, for nodes, pods and containers, through its Summary API and its /metrics/cadvisor endpoint. Part 3 does the same for memory.

Lab: 19-metrics-and-psi covers both halves. Check what you built with make check LAB=19.

#What's next

CPU fails slowly: it makes you wait. Memory fails all at once. Part 3: Memory covers OOM kills, why a pod can be killed while the dashboard says it had room to spare, what happens when a whole node runs short, the memory numbers to watch in production, and resizing a running pod.