[bug] Kubernetes / Views / Nodes no data

[trungdlp]

Describe the bug

Should change

sum(container_memory_working_set_bytes{node="$node", image!="", cluster="$cluster"}) by (pod)

to

sum(container_memory_working_set_bytes{instance="$node", image!="", cluster="$cluster"}) by (pod)

Dashboard: Kubernetes / Views / Nodes

Edit panel

How to reproduce?

No response

Expected behavior

No response

Additional context

No response

[filipisaci]

Dashboard works on AKS but not on EKS - cluster label issue

Environment:

• kube-prometheus-stack chart: v76.5.1
• Grafana dashboard: k8s-views-nodes (ID 15759)
• Cloud providers: Azure AKS (working) vs AWS EKS (broken)
• Prometheus Operator with kube-state-metrics, node-exporter, kubelet/cAdvisor

Issue:
The dashboard works perfectly on our AKS cluster but fails on EKS with empty "CPU usage by Pod" and "Memory usage by Pod" panels.

Root cause:
The dashboard queries filter by cluster="$cluster" in cAdvisor metrics (e.g., container_cpu_usage_seconds_total), but kube-prometheus-stack does not inject the cluster label into cAdvisor metrics by default.

Why it works on AKS but not EKS:
We suspect our AKS setup had custom relabelings or used Azure Monitor's managed Prometheus, which automatically injects cluster labels. On EKS, using the stock kube-prometheus-stack chart, the cluster label is missing from container_* metrics.

Attempted solutions:

1. ✅ Added kubelet.serviceMonitor.relabelings to inject cluster label → Only works for endpoint 0 (resource metrics), not for cAdvisor endpoint
2. ❌ Tried kubelet.serviceMonitor.metricRelabelings → Not supported for per-endpoint configuration in chart v76.5.1 (nor in v79.5.0)
3. ❌ Tried prometheusSpec.metricRelabelings → Field doesn't exist in the chart schema

Current workaround:
Created a local copy of the dashboard and removed the cluster="$cluster" filter from all cAdvisor queries:

# Before
sum(rate(container_cpu_usage_seconds_total{node="$node", image!="", cluster="$cluster"}[$__rate_interval])) by (pod)

# After (working)
sum(rate(container_cpu_usage_seconds_total{node="$node", image!=""}[$__rate_interval])) by (pod)