Bug (upgrade): Support for cgroupv2 in coder app when pulling stats

[bjornrobertsson]

Changes in cgroupv2 cause coder stat to fail for CPU and memory

$ coder stat mem
Encountered an error running "coder stat mem", see "coder stat mem --help" for more information
error: read memory total: read /sys/fs/cgroup/memory/memory.limit_in_bytes: open /sys/fs/cgroup/memory/memory.limit_in_bytes: no such file or directory

This happens when cgroupv2 is active, and you're inside a container.

The default location of these for cgroupv2 has changed to, i.e.: /sys/fs/cgroup/memory.max BUT this is only true of a 'native' execution, i.e. a VM/host/node. When you run inside a container (read as a pod in k8s) the files will be split up, i.e. /sys/fs/cgroup/system.slice/memory.max or address with some 'obscure' identifiers.

The current agents are therefore likely to show up as failing due to not finding the right resource.

[bjornrobertsson]

This is the content on the Azure Envbox Image:

root@coder-bjorn-attempt2:/sys/fs/cgroup# ls
aks.slice               cgroup.procs            cpu.pressure           dev-hugepages.mount  io.stat           memory.stat                    sys-kernel-debug.mount
cgroup.controllers      cgroup.stat             cpu.stat               dev-mqueue.mount     kubepods.slice    proc-sys-fs-binfmt_misc.mount  sys-kernel-tracing.mount
cgroup.max.depth        cgroup.subtree_control  cpuset.cpus.effective  init.scope           memory.numa_stat  sys-fs-fuse-connections.mount  system.slice
cgroup.max.descendants  cgroup.threads          cpuset.mems.effective  io.pressure          memory.pressure   sys-kernel-config.mount        user.slice
and there we have the whole thing move to system and user.slice subfolder:
memory.current       memory.events.local  memory.low           memory.min           memory.oom.group     memory.stat          memory.swap.events   memory.swap.max
memory.events        memory.high          memory.max           memory.numa_stat     memory.pressure      memory.swap.current  memory.swap.high
root@coder-bjorn-attempt2:/sys/fs/cgroup/user.slice# cat /sys/fs/cgroup/system.slice/memory.max
max
root@coder-bjorn-attempt2:/sys/fs/cgroup/user.slice# cat /sys/fs/cgroup/system.slice/memory.stat
anon 288096256
file 5922144256
kernel_stack 3489792
pagetables 4759552
...

There is something not right about the coder cli depending on the underlying method. Can we do additional testing for these variants?

Another slight difference with the cgroupv2, if we expect a number, memory.max may not even be a number:

root@4318e07723c6:/sys/fs/cgroup# cat memory.max
max
root@4318e07723c6:/sys/fs/cgroup# cat memory.min
0

it is sometimes (azure workspace):

podman@coder-4ce38369-ecdf-49df-a3bb-4a1190b3f7d0:/sys/fs/cgroup$ cat memory.max 
2147483648

[github-actions]

❌ No matching Coder user found for GitHub user @bjornrobertsson. Please connect your GitHub account with Coder and try again: https://dev.coder.com/settings/external-auth

[bjornrobertsson]

Moving this to clistat since the cgroupv2 is working here although Envbox still has issues.

CPU: 0.095/2 cores (5%)
Memory: 1.62/15.4 GiB (11%)
Disk: 32.3/125 GiB (26%)
2025/05/28 12:19:24 get total cpu: 2 errors occurred:
	* get cpu period: read /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us: open /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us: no such file or directory
	* get cpu period: read /sys/fs/cgroup/cpu/cpu.cfs_period_us: open /sys/fs/cgroup/cpu/cpu.cfs_period_us: no such file or directory

exit status 1

With the k8s resources present:

root@coder-bjorn-attempt2:/sys/fs/cgroup# ls
aks.slice               cgroup.stat             cpuset.cpus.effective  init.scope        memory.pressure                sys-kernel-debug.mount
cgroup.controllers      cgroup.subtree_control  cpuset.mems.effective  io.pressure       memory.stat                    sys-kernel-tracing.mount
cgroup.max.depth        cgroup.threads          dev-hugepages.mount    io.stat           proc-sys-fs-binfmt_misc.mount  system.slice
cgroup.max.descendants  cpu.pressure            dev-mqueue.mount       kubepods.slice    sys-fs-fuse-connections.mount  user.slice
cgroup.procs            cpu.stat                docker                 memory.numa_stat  sys-kernel-config.mount

There is no (cfs) entries here at all

[bjornrobertsson]

Clarifying the testing/error platform:

Edge case shows up on a small set of test platforms, Azure k8s cluster:

(Coder 2.20.2, Ubuntu 24.04 on AKS, 'Devcontainer')

• ghcr.io/coder/envbuilder:1.1.0
• using enterprise-base, with podman tools installed

root ➜ / $ ls /sys/fs/cgroup/
aks.slice               cgroup.procs            cpu.pressure           dev-hugepages.mount  io.stat           memory.stat                    sys-kernel-debug.mount
cgroup.controllers      cgroup.stat             cpu.stat               dev-mqueue.mount     kubepods.slice    proc-sys-fs-binfmt_misc.mount  sys-kernel-tracing.mount
cgroup.max.depth        cgroup.subtree_control  cpuset.cpus.effective  init.scope           memory.numa_stat  sys-fs-fuse-connections.mount  system.slice
cgroup.max.descendants  cgroup.threads          cpuset.mems.effective  io.pressure          memory.pressure   sys-kernel-config.mount        user.slice
root ➜ / $ ./tmp/coder.OXppCK/coder stat mem
Encountered an error running "coder stat mem", see "coder stat mem --help" for more information
error: read memory total: read /sys/fs/cgroup/memory/memory.limit_in_bytes: open /sys/fs/cgroup/memory/memory.limit_in_bytes: no such file or directory

Expected outcome:

(Coder 2.22.0, Ubuntu 24.04 on RKE1 Cluster with Ubuntu 20.04 nodes, 'Standard k8s Workspace')

coder@coder-2f1d39a2-791b-4edb-a07e-04e503fdfcd7-747f59dfc6-rkc46:~$ coder stat mem
0.252/2 GiB (13%)
coder@coder-2f1d39a2-791b-4edb-a07e-04e503fdfcd7-747f59dfc6-rkc46:~$ coder stat cpu
0.002/2 cores (0%)

(Coder 2.20.2, Ubuntu 24.04 on AKS, AKSAzureLinux-V2gen2-202505.14.0, 'Envbox')

root@EnvboxTestingShm:/# /home/coder/.coder/coder stat mem
0.373/7.45 GiB (5%)

[bjornrobertsson]

systemd-enabled-template.txt

I used this template to enable systemd to express the errors (since systemd causes the slice behaviour)

[fh-cris-forno]

+1 , would like to see support on this as newer AMIs on the Amazon side are phasing cgroup v1 out by default.

[brent-at-aam]

I've implemented a couple of shell functions in our deployment to pull cpu and memory stats for a container. I'll share it here in case it's helpful for the Coder team to implement a fix in clistat.

We're using Coder in EKS with Bottlerocket as the node OS. For our workspaces, we let developers choose a CPU/Memory configuration which we use to set the resource requirements for the container. If a user selects 4 cpu and 16 GiB memory, we set the memory request and limit to 16 GiB and the cpu request to 4 while leaving the limit unset. This lets users burst a bit but has an effect on reporting the total cpu (more on this below).

Our team saw the same error when running coder stat mem/cpu

❯ coder stat mem   
Encountered an error running "coder stat mem", see "coder stat mem --help" for more information
error: read memory total: read /sys/fs/cgroup/memory/memory.limit_in_bytes: open /sys/fs/cgroup/memory/memory.limit_in_bytes: no such file or directory

The workaround is to adjust the base path used to lookup the relevant cgroups files. To get the correct path, we extract the slice for the current container from /proc/self/cgroup

cat /proc/self/cgroup 
0::/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea12d685_4626_403c_8ea2_f630e0bc3aa3.slice/cri-containerd-13c0cba08d3c5e64914c8d6c7e5f14951ecfb6599374389cee5f47717bf4d4fe.scope

In this example our base path for cgroups stats should be adjusted to:

/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea12d685_4626_403c_8ea2_f630e0bc3aa3.slice/cri-containerd-13c0cba08d3c5e64914c8d6c7e5f14951ecfb6599374389cee5f47717bf4d4fe.scope

The memory calculation is mostly the same (I'm not substracting inactive_file from the current).

#!/bin/sh

memory_stats () 
{ 
    base_path="/sys/fs/cgroup/$(cat /proc/self/cgroup | grep 'kubepods.slice' | cut -d : -f 3)";
    current_path="$base_path/memory.current";
    max_path="$base_path/memory.max";
    current=$(cat "$current_path");
    max=$(cat "$max_path");
    percent=$(echo "x=($current / $max) * 100; scale=2; x / 1" | bc -l);
    current_gb=$(echo "x=$current / 1024 / 1024 / 1024; scale=2; x / 1" | bc -l);
    max_gb=$(echo "x=$max / 1024 / 1024 / 1024; scale=2; x / 1" | bc -l);
    echo "$current_gb/$max_gb GiB ($percent%)"
}

❯ coder-memory-stats 
3.00/16.00 GiB (18.78%)

For cpu, I think the total cpu calculation might be different if we were setting a cpu limit, but this is what we came up with for our scenario. We derive the cpu count from the ratio of the cpu weight assigned to our container vs the cpu weight for all containers on the node, multiplied by the total cpu on the node.

cpu_stats () 
{ 
    base_path="/sys/fs/cgroup/$(cat /proc/self/cgroup | grep 'kubepods.slice' | cut -d : -f 3)";
    stat_path="$base_path/cpu.stat";
    max_path="$base_path/cpu.max";
    weight_path="$base_path/cpu.weight";
    available_weight_path="/sys/fs/cgroup/kubepods.slice/cpu.weight";
    container_cpu_weight=$(cat "$weight_path");
    available_cpu_weight=$(cat "$available_weight_path");
    host_cpus=$(nproc --all);
    cpus=$(echo "p=$container_cpu_weight / $available_cpu_weight; x=p * $host_cpus; scale=2; x / 1" | bc -l);
    period=$(awk -F " " '{ print $2 }' "$max_path");
    tstart=$(date +%s%N);
    ustart=$(awk -F " " '$1 == "usage_usec" { print $2 }' "$stat_path");
    sleep 5;
    tstop=$(date +%s%N);
    ustop=$(awk -F " " '$1 == "usage_usec" { print $2 }' "$stat_path");
    usage=$(echo "x=(($ustop - $ustart) * 1000) / ($tstop - $tstart); scale=2; x / 1" | bc -l);
    percent=$(echo "x=($usage / $cpus) * 100; scale=2; x / 1" | bc -l);
    echo "$usage/$cpus ($percent%)"
}

❯ coder-cpu-stats 
.03/4.03 (.74%)

I hope this information is helpful!

[DanielleMaywood]

@brent-at-aam I recently refactored our logic with the approach you described in 50cfe85

This fix is now available in

• Coder v2.26.2
• Coder v2.27.1