How do you handle mounts coming and going?

[keegancsmith]

Hey, I authored a similiar project https://github.com/keegancsmith/kubernetes-disk-exporter

I have one major issue, as mounts come and go the rootfs mount doesn't see the new mounts. Does your implementation have the same issue? I see you set privilege: true, which my k8s daemonset is missing.

[joar]

While I have not tested the specific case of

1. Exporter starts on
2. Pod with new PVC/PD scheduled on $node
3. Ensure new PVC/PD present in exporter metrics

I have not found any such issue.

---

Regarding privileged: true, I can't remember exactly the issue. I think it was that I could not view, or could not access the mounts of other users if I did not have the equivalent of root privileges. I tried with capabilities, but could not find any that would allow me to access the mounts. I scribbled something down at

k8s-pv-disk-usage-exporter/example/daemonset.yaml

Lines 18 to 22 in 134c9eb

	# XXX: If this is not set, disk-usage-exporter will not be able to
	# find any mounted PVs.
	# There might be a better, more fine-tuned setting to use, but I have
	# not yet found one.
	privileged: true

[keegancsmith]

@joar thanks for the response. As reference this is the issue I have for this keegancsmith/kubernetes-disk-exporter#1
To see if you are affected try getting a pod with a pv disk to move nodes (ie delete a pod attached to a replicaset).

I will try setting privileged true on a cluster at work to see if it resolves it.

Edit: Adding privileged=true didn't help in my project. But another difference I see is you mount the rootfs, while I just mount /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts. If I mount the rootfs I may have access to changing mounts. Will try that later.

[joar]

@keegancsmith For each metric scrape, I list all available mountpoints:

k8s-pv-disk-usage-exporter/disk_usage_exporter/collect/partitions.py

Lines 24 to 33 in 134c9eb

	async def get_partitions(ctx: Context, *, loop=None) -> List[Partition]:
	loop = loop or asyncio.get_event_loop()
	_partitions = await loop.run_in_executor(
	ctx.executor,
	psutil.disk_partitions
	) # type: List[psutil._common.sdiskpart]
	return [
	Partition(*_partition)
	for _partition in _partitions
	]

using psutil.disk_partitions() (linux version)

I then filter

k8s-pv-disk-usage-exporter/disk_usage_exporter/collect/__init__.py

Lines 187 to 188 in 134c9eb

	def filter_pv(partition: Partition) -> bool:
	return get_pv_name(partition) is not None

the mountpoints

k8s-pv-disk-usage-exporter/disk_usage_exporter/collect/partitions.py

Lines 36 to 41 in 134c9eb

	def get_pv_name(partition: Partition) -> Optional[str]:
	match = MOUNTPOINT_PV_RE.match(partition.mountpoint)
	if match is None:
	return
	return match.group('pv_name')

to only include paths that look like

k8s-pv-disk-usage-exporter/disk_usage_exporter/collect/partitions.py

Lines 44 to 63 in 134c9eb

	# Example:
	# /rootfs/home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/
	# 4bb9d022-5a63-11e7-ba69-42010af0012c/volumes/kubernetes.io~gce-pd/
	# pvc-4bb92cb4-5a63-11e7-ba69-42010af0012c
	MOUNTPOINT_PV_RE = re.compile(r'''
	^
	(?P<prefix>
	.*
	/kubelet/pods/
	.*?
	volumes/kubernetes.io
	# A tilde, not a hyphen
	~
	gce-pd/
	)
	(?P<pv_name>
	[^/]+
	)
	$
	''', re.VERBOSE)

I also extract the PV name from the mountpoint

k8s-pv-disk-usage-exporter/disk_usage_exporter/collect/partitions.py

Lines 36 to 41 in 134c9eb

	def get_pv_name(partition: Partition) -> Optional[str]:
	match = MOUNTPOINT_PV_RE.match(partition.mountpoint)
	if match is None:
	return
	return match.group('pv_name')

Finally to get the FS usage for each mountpoint, I run psutil.disk_usage()

k8s-pv-disk-usage-exporter/disk_usage_exporter/collect/__init__.py

Lines 27 to 32 in e6ec2ee

	def values_from_path(path: str, labels: Optional[Dict]=None) -> List[MetricValue]:
	disk_usage = psutil.disk_usage(path)
	labels = labels or dict(
	path=path,
	)

I also query Kubernetes for the PV and PVC's name and labels in order to attach them to the mountpoint FS metrics.