Kubernetes Devcontainer Template, Issues with CPU & Memory Agent Metadata

[greg-the-coder]

I've encountered issues when using the Kubernetes Devcontainer template (https://github.com/coder/registry/blob/main/registry/coder/templates/kubernetes-devcontainer/main.tf) on AWS EKS/Linux Nodes, and SUSE/Rancher RKE Nodes where the Coder Agent CPU and Memory Metadata won't display properly. What I discovered is that the "coder stat cpu" and "coder stat mem" commands error out.

Here are examples from running the command in Kubernetes Devcontainer workspaces on AWS EKS using auto-mode and bottlerocket Node group:

*** cpu ***
coder@coder-7aa67e3d-df1b-4074-9379-e44ff309fcec-866f5bbb4c-2rk6j:/workspaces$ coder stat cpu
Encountered an error running "coder stat cpu", see "coder stat cpu --help" for more information
error: get total cpu: 2 errors occurred:
        * get cpu period: read /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us: open /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us: no such file or directory
        * get cpu period: read /sys/fs/cgroup/cpu/cpu.cfs_period_us: open /sys/fs/cgroup/cpu/cpu.cfs_period_us: no such file or directory

*** mem ***
coder@coder-7aa67e3d-df1b-4074-9379-e44ff309fcec-866f5bbb4c-2rk6j:/workspaces$ coder stat mem
Encountered an error running "coder stat mem", see "coder stat mem --help" for more information
error: read memory total: read /sys/fs/cgroup/memory/memory.limit_in_bytes: open /sys/fs/cgroup/memory/memory.limit_in_bytes: no such file or directory

[pakwin9133]

Got the same issue in rke2 with sysbox

[DanielleMaywood]

Got the same issue in rke2 with sysbox

@pakwin9133 Can you confirm the version of Coder you're running? We've recently introduced a fix that might resolve this in clistat v1.1.1. The fix should be present in Coder 2.26.2, 2.27.1 and 2.28.0.

[pakwin9133]

i tried both 2.26.2 and 2.27.1. They are not working well. 2.26.1 (my current version) has no issue.

[greg-the-coder]

@DanielleMaywood I just ran a successful test using 2.26.3 in one of my Coder demo deployments on EKS/Bottlerocket, where I originally observed the issue. Have not had a chance to try on RKE though, I don't have any environments available for that at the moment, but might be able to check that out next week.

[greg-the-coder]

@DanielleMaywood was able to find and upgrade my SUSE/Rancher demo environment that uses RKE2 to 2.28.0, and that test was successful also.

[pakwin9133]

I also have another coder (2.27.3) which doesn't have any issue:

####################################################################
$ coder --version
Coder v2.27.3+800dd9c Thu Oct 30 19:32:36 UTC 2025
coder/coder@800dd9c

Slim build of Coder, does not support the server subcommand.
$ coder stat cpu
0.031/8 cores (0%)
$ coder stat mem
0.251/8 GiB (3%)
$ ls /sys/fs/cgroup/init.scope
ls: cannot access '/sys/fs/cgroup/init.scope': No such file or directory
$
####################################################################

As you can see, clistat works fine even /sys/fs/cgroup/init.scope doesn't exist.
It seems that clistat has some incorrect logic to determine which path/file should be checked.

[pakwin9133]

@DanielleMaywood , thanks for the update. Your last commit 9017018 resolves the container cpu stat issue. Could you also consider to change the memory implementation as well ??

//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
func (s cgroupV2Statter) memoryCurrentBytes() (*int64, error) {
	memoryUsagePath := filepath.Join(s.path, cgroupV2MemoryUsageBytes)

	maxUsageBytes, err := readInt64(s.fs, memoryUsagePath)
	if err != nil {
		if !errors.Is(err, strconv.ErrSyntax) && !errors.Is(err, fs.ErrNotExist) {
			return nil, xerrors.Errorf("read memory usage: %w", err)
		}

		if s.parent != nil {
			result, err := s.parent.memoryCurrentBytes()
			if err != nil {
				return nil, xerrors.Errorf("read parent memory max: %w", err)
			}
			return result, nil
		}

		// We have no parent, and no max memory limit, so there is no memory limit.
		return nil, nil
	}

	return ptr.To(maxUsageBytes), nil
}

func (s cgroupV2Statter) memoryStatBytes() (*int64, error) {
	memoryStatPath := filepath.Join(s.path, cgroupV2MemoryStat)

	maxStatBytes, err := readInt64Prefix(s.fs, memoryStatPath, "inactive_file")
	if err != nil {
		if !errors.Is(err, strconv.ErrSyntax) && !errors.Is(err, fs.ErrNotExist) {
			return nil, xerrors.Errorf("read memory usage: %w", err)
		}

		if s.parent != nil {
			result, err := s.parent.memoryStatBytes()
			if err != nil {
				return nil, xerrors.Errorf("read parent memory max: %w", err)
			}
			return result, nil
		}

		// We have no parent, and no max memory limit, so there is no memory limit.
		return nil, nil
	}

	return ptr.To(maxStatBytes), nil
}

func (s cgroupV2Statter) memory(p Prefix) (*Result, error) {
	// https://docs.kernel.org/6.17/admin-guide/cgroup-v2.html#memory-interface-files
	r := &Result{
		Unit:   "B",
		Prefix: p,
	}
	if total, err := s.memoryMaxBytes(); err != nil {
		return nil, xerrors.Errorf("read memory total: %w", err)
	} else {
		r.Total = total
	}

	currUsageBytes, err := s.memoryCurrentBytes()
	if err != nil {
		return nil, xerrors.Errorf("read memory usage: %w", err)
	}

	inactiveFileBytes, err := s.memoryStatBytes()
	if err != nil {
		return nil, xerrors.Errorf("read memory stats: %w", err)
	}

	r.Used = float64(*currUsageBytes - *inactiveFileBytes)
	return r, nil
}
//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

[DanielleMaywood]

@pakwin9133 Thanks for confirming the fix works. I'll get a change in to fix the memory statter on Monday.

[pakwin9133]

@DanielleMaywood , the commit 0f14d87 works very well. Thanks for the update. May I know when it will be tagged and included into coderd ??

[DanielleMaywood]

@pakwin9133 I will get it tagged tomorrow (Thursday 20th November) and merged into coder/coder tomorrow as well. This means it will be included in the upcoming Coder 2.29 (to be released near the beginning of December).

[DanielleMaywood]

@pakwin9133 I've merged an update on the dependency on coder/clistat in coder/coder today. Providing no unforeseen issues it should be in the upcoming 2.29 release.