Cannot authenticate agains proxmox when using username and password

[justinas-b]

/kind bug

What steps did you take and what happened:
I am using below variables to generate manifests:

export PROXMOX_URL="https://pve.mydomain.net/api2/json"
export PROXMOX_PASSWORD='MyPa$$w0rd'
export PROXMOX_USERNAME='svc-pve-01@MYDOMAIN.NET'
...
clusterctl generate cluster capi-mgmt \
    --control-plane-machine-count=1 \
    --infrastructure=proxmox:v0.3.5 \
    --worker-machine-count 1 \
    --kubernetes-version v1.27.8 \
    --config https://raw.githubusercontent.com/k8s-proxmox/cluster-api-provider-proxmox/main/clusterctl.yaml 

However capi fails to authenticate agains proxmox:

❯ k logs deployments/cappx-controller-manager manager --follow --since=3h
I0522 15:21:43.309427       1 listener.go:44] "controller-runtime/metrics: Metrics server is starting to listen" addr="127.0.0.1:8080"
I0522 15:21:43.310046       1 scheduler.go:45] "load plugin config: {map[CPUOvercommit:{false map[]} MemoryOvercommit:{false map[]}] map[] map[]}"
I0522 15:21:43.310088       1 main.go:139] "setup: starting manager"
I0522 15:21:43.310382       1 internal.go:360] "Starting server" kind="health probe" addr="[::]:8081"
I0522 15:21:43.310438       1 server.go:50] "starting server" path="/metrics" kind="metrics" addr="127.0.0.1:8080"
I0522 15:21:43.310507       1 leaderelection.go:245] attempting to acquire leader lease cappx-system/36404136.cluster.x-k8s.io...
I0522 15:22:00.168821       1 leaderelection.go:255] successfully acquired lease cappx-system/36404136.cluster.x-k8s.io
I0522 15:22:00.170319       1 controller.go:177] "Starting EventSource" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" source="kind source: *v1beta1.ProxmoxMachine"
I0522 15:22:00.170374       1 controller.go:185] "Starting Controller" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine"
I0522 15:22:00.170853       1 controller.go:177] "Starting EventSource" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" source="kind source: *v1beta1.ProxmoxCluster"
I0522 15:22:00.170905       1 controller.go:185] "Starting Controller" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster"
I0522 15:22:00.278039       1 controller.go:219] "Starting workers" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" worker count=1
I0522 15:22:00.278040       1 controller.go:219] "Starting workers" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" worker count=1
I0522 15:22:00.493614       1 proxmoxcluster_controller.go:108] "Reconciling ProxmoxCluster" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=a6e42b05-14fb-4f54-8a8a-876916820d23
I0522 15:22:00.494352       1 reconcile.go:20] "Reconciling storage" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=a6e42b05-14fb-4f54-8a8a-876916820d23
I0522 15:22:00.515179       1 proxmoxmachine_controller.go:144] "Reconciling ProxmoxMachine" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/capi-mgmt-md-0-z8rlk-sjxvj" namespace="default" name="capi-mgmt-md-0-z8rlk-sjxvj" reconcileID=4604043f-8556-4577-8133-ddefbf46a400
I0522 15:22:00.928273       1 reconcile.go:42] "storage %s not found. it will be created" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=a6e42b05-14fb-4f54-8a8a-876916820d23
I0522 15:22:00.998130       1 reconcile.go:24] "Reconciling instance" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/capi-mgmt-md-0-z8rlk-sjxvj" namespace="default" name="capi-mgmt-md-0-z8rlk-sjxvj" reconcileID=4604043f-8556-4577-8133-ddefbf46a400
I0522 15:22:00.998256       1 reconcile.go:105] "instance does not have providerID yet" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/capi-mgmt-md-0-z8rlk-sjxvj" namespace="default" name="capi-mgmt-md-0-z8rlk-sjxvj" reconcileID=4604043f-8556-4577-8133-ddefbf46a400
I0522 15:22:00.998479       1 reconcile.go:89] "instance wasn't found. new instance will be created" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/capi-mgmt-md-0-z8rlk-sjxvj" namespace="default" name="capi-mgmt-md-0-z8rlk-sjxvj" reconcileID=4604043f-8556-4577-8133-ddefbf46a400
I0522 15:22:00.998501       1 qemu.go:21] "Reconciling QEMU" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/capi-mgmt-md-0-z8rlk-sjxvj" namespace="default" name="capi-mgmt-md-0-z8rlk-sjxvj" reconcileID=4604043f-8556-4577-8133-ddefbf46a400
I0522 15:22:00.998514       1 qemu.go:58] "getting qemu from vmid" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/capi-mgmt-md-0-z8rlk-sjxvj" namespace="default" name="capi-mgmt-md-0-z8rlk-sjxvj" reconcileID=4604043f-8556-4577-8133-ddefbf46a400
I0522 15:22:00.998759       1 scheduler.go:173] "Start Running Scheduler" Name="qemu-scheduler"
I0522 15:22:00.998997       1 scheduler.go:196] "getting next qemu from scheduling queue" Name="qemu-scheduler"
E0522 15:22:01.093892       1 proxmoxcluster_controller.go:123] "Reconcile error" err="403 - 403 Forbidden - {\"data\":null}" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=a6e42b05-14fb-4f54-8a8a-876916820d23
E0522 15:22:01.095034       1 controller.go:324] "Reconciler error" err="403 - 403 Forbidden - {\"data\":null}" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=a6e42b05-14fb-4f54-8a8a-876916820d23
I0522 15:22:01.109755       1 proxmoxcluster_controller.go:108] "Reconciling ProxmoxCluster" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=a9592575-f8cd-4127-8095-dea292f983da
I0522 15:22:01.110399       1 reconcile.go:20] "Reconciling storage" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=a9592575-f8cd-4127-8095-dea292f983da
I0522 15:22:01.163051       1 reconcile.go:42] "storage %s not found. it will be created" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=a9592575-f8cd-4127-8095-dea292f983da
E0522 15:22:01.228152       1 proxmoxcluster_controller.go:123] "Reconcile error" err="403 - 403 Forbidden - {\"data\":null}" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=a9592575-f8cd-4127-8095-dea292f983da
E0522 15:22:01.228795       1 controller.go:324] "Reconciler error" err="403 - 403 Forbidden - {\"data\":null}" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=a9592575-f8cd-4127-8095-dea292f983da
I0522 15:22:01.244634       1 proxmoxcluster_controller.go:108] "Reconciling ProxmoxCluster" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=1aea7b46-38ac-4f19-b093-fc0a7ad64bfd
I0522 15:22:01.244857       1 reconcile.go:20] "Reconciling storage" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=1aea7b46-38ac-4f19-b093-fc0a7ad64bfd
I0522 15:22:01.292835       1 reconcile.go:42] "storage %s not found. it will be created" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=1aea7b46-38ac-4f19-b093-fc0a7ad64bfd
E0522 15:22:01.347866       1 proxmoxcluster_controller.go:123] "Reconcile error" err="403 - 403 Forbidden - {\"data\":null}" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=1aea7b46-38ac-4f19-b093-fc0a7ad64bfd
E0522 15:22:01.348718       1 controller.go:324] "Reconciler error" err="403 - 403 Forbidden - {\"data\":null}" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=1aea7b46-38ac-4f19-b093-fc0a7ad64bfd
I0522 15:22:01.365569       1 qemu.go:68] "creating qemu" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/capi-mgmt-md-0-z8rlk-sjxvj" namespace="default" name="capi-mgmt-md-0-z8rlk-sjxvj" reconcileID=4604043f-8556-4577-8133-ddefbf46a400
I0522 15:22:01.365633       1 storage.go:15] "ensuring storage is available" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/capi-mgmt-md-0-z8rlk-sjxvj" namespace="default" name="capi-mgmt-md-0-z8rlk-sjxvj" reconcileID=4604043f-8556-4577-8133-ddefbf46a400
I0522 15:22:01.365766       1 storage.go:40] "finding available storage" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/capi-mgmt-md-0-z8rlk-sjxvj" namespace="default" name="capi-mgmt-md-0-z8rlk-sjxvj" reconcileID=4604043f-8556-4577-8133-ddefbf46a400
I0522 15:22:01.378377       1 proxmoxcluster_controller.go:108] "Reconciling ProxmoxCluster" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=caf0200c-6aeb-41af-8165-975b50891f51
I0522 15:22:01.378890       1 reconcile.go:20] "Reconciling storage" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=caf0200c-6aeb-41af-8165-975b50891f51
I0522 15:22:01.413743       1 qemu.go:75] "making qemu spec" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/capi-mgmt-md-0-z8rlk-sjxvj" namespace="default" name="capi-mgmt-md-0-z8rlk-sjxvj" reconcileID=4604043f-8556-4577-8133-ddefbf46a400
I0522 15:22:01.413952       1 scheduler.go:254] "adding qemu to scheduler queue" Name="qemu-scheduler" qemu="capi-mgmt-md-0-z8rlk-sjxvj"
I0522 15:22:01.414031       1 scheduler.go:203] "scheduling qemu" Name="qemu-scheduler" qemu="capi-mgmt-md-0-z8rlk-sjxvj"
I0522 15:22:01.414113       1 scheduler.go:273] "finding proxmox node matching qemu" Name="qemu-scheduler" qemu="capi-mgmt-md-0-z8rlk-sjxvj"
I0522 15:22:01.433062       1 reconcile.go:42] "storage %s not found. it will be created" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=caf0200c-6aeb-41af-8165-975b50891f51
E0522 15:22:01.478752       1 proxmoxcluster_controller.go:123] "Reconcile error" err="403 - 403 Forbidden - {\"data\":null}" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=caf0200c-6aeb-41af-8165-975b50891f51
E0522 15:22:01.479328       1 controller.go:324] "Reconciler error" err="403 - 403 Forbidden - {\"data\":null}" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=caf0200c-6aeb-41af-8165-975b50891f51
I0522 15:22:01.490278       1 scheduler.go:320] "filtering proxmox node" Name="qemu-scheduler" qemu="capi-mgmt-md-0-z8rlk-sjxvj"
I0522 15:22:01.527408       1 proxmoxcluster_controller.go:108] "Reconciling ProxmoxCluster" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=41c9d6ab-3eb8-45ec-925f-66eec21886a0
I0522 15:22:01.527741       1 reconcile.go:20] "Reconciling storage" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=41c9d6ab-3eb8-45ec-925f-66eec21886a0
I0522 15:22:01.581912       1 reconcile.go:42] "storage %s not found. it will be created" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=41c9d6ab-3eb8-45ec-925f-66eec21886a0
E0522 15:22:01.648182       1 proxmoxcluster_controller.go:123] "Reconcile error" err="403 - 403 Forbidden - {\"data\":null}" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=41c9d6ab-3eb8-45ec-925f-66eec21886a0
E0522 15:22:01.649026       1 controller.go:324] "Reconciler error" err="403 - 403 Forbidden - {\"data\":null}" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=41c9d6ab-3eb8-45ec-925f-66eec21886a0
I0522 15:22:01.741568       1 proxmoxcluster_controller.go:108] "Reconciling ProxmoxCluster" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=341879fb-1939-4f2e-8a0a-9077a18b47f4
I0522 15:22:01.742177       1 reconcile.go:20] "Reconciling storage" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=341879fb-1939-4f2e-8a0a-9077a18b47f4
I0522 15:22:01.792431       1 reconcile.go:42] "storage %s not found. it will be created" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=341879fb-1939-4f2e-8a0a-9077a18b47f4
I0522 15:22:02.624244       1 scheduler.go:343] "scoring proxmox node" Name="qemu-scheduler" qemu="capi-mgmt-md-0-z8rlk-sjxvj"
E0522 15:22:03.554995       1 proxmoxcluster_controller.go:123] "Reconcile error" err="403 - 403 Forbidden - {\"data\":null}" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=341879fb-1939-4f2e-8a0a-9077a18b47f4
E0522 15:22:03.556002       1 controller.go:324] "Reconciler error" err="403 - 403 Forbidden - {\"data\":null}" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=341879fb-1939-4f2e-8a0a-9077a18b47f4
I0522 15:22:03.725758       1 proxmoxcluster_controller.go:108] "Reconciling ProxmoxCluster" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=11602d6c-be71-4536-ad2b-88eedd538f9d
I0522 15:22:03.726315       1 reconcile.go:20] "Reconciling storage" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=11602d6c-be71-4536-ad2b-88eedd538f9d
I0522 15:22:05.563880       1 reconcile.go:42] "storage %s not found. it will be created" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=11602d6c-be71-4536-ad2b-88eedd538f9d
E0522 15:22:07.560205       1 proxmoxcluster_controller.go:123] "Reconcile error" err="403 - 403 Forbidden - {\"data\":null}" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=11602d6c-be71-4536-ad2b-88eedd538f9d
E0522 15:22:07.560741       1 controller.go:324] "Reconciler error" err="403 - 403 Forbidden - {\"data\":null}" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=11602d6c-be71-4536-ad2b-88eedd538f9d
I0522 15:22:07.890023       1 proxmoxcluster_controller.go:108] "Reconciling ProxmoxCluster" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=c190bc63-ce4c-4ee5-922b-229287611edd
I0522 15:22:07.890435       1 reconcile.go:20] "Reconciling storage" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=c190bc63-ce4c-4ee5-922b-229287611edd
I0522 15:22:09.550213       1 reconcile.go:42] "storage %s not found. it will be created" controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/capi-mgmt" namespace="default" name="capi-mgmt" reconcileID=c190bc63-ce4c-4ee5-922b-229287611edd
E0522 15:22:10.633383       1 runtime.go:79] Observed a panic: "integer divide by zero" (runtime error: integer divide by zero)
goroutine 225 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x11f0140?, 0x22faf10})
	/go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/runtime/runtime.go:75 +0x7c
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x20?})
	/go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/runtime/runtime.go:49 +0x78
panic({0x11f0140, 0x22faf10})
	/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/k8s-proxmox/cluster-api-provider-proxmox/cloud/scheduler/plugins/noderesource.(*NodeResource).Score(_, {_, _}, _, {0x0, {0x0, 0x0}, {0x13f660b, 0x9}, {0x0, ...}, ...}, ...)
	/workspace/cloud/scheduler/plugins/noderesource/node_resrouce.go:29 +0x64
github.com/k8s-proxmox/cluster-api-provider-proxmox/cloud/scheduler.(*Scheduler).RunScorePlugins(_, {_, _}, _, {0x0, {0x0, 0x0}, {0x13f660b, 0x9}, {0x0, ...}, ...}, ...)
	/workspace/cloud/scheduler/scheduler.go:353 +0x32c
github.com/k8s-proxmox/cluster-api-provider-proxmox/cloud/scheduler.(*Scheduler).SelectNode(_, {_, _}, {0x0, {0x0, 0x0}, {0x13f660b, 0x9}, {0x0, 0x0}, ...})
	/workspace/cloud/scheduler/scheduler.go:291 +0x19c
github.com/k8s-proxmox/cluster-api-provider-proxmox/cloud/scheduler.(*Scheduler).ScheduleOne(0x400058c5a0, {0x16724c8?, 0x40001fc4b0?})
	/workspace/cloud/scheduler/scheduler.go:210 +0x244
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
	/go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/backoff.go:259 +0x2c
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x40004dfe28?)
	/go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/backoff.go:226 +0x40
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x165c108, 0x4000b46180}, 0x1, 0x4000590540)
	/go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/backoff.go:227 +0x90
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x40001fc4b0?, 0x0, 0x0, 0x0?, 0x140a5a1?)
	/go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/backoff.go:204 +0x80
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x16724c8, 0x40001fc4b0}, 0x40004dff90, 0x140a5a1?, 0x17?, 0x0?)
	/go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/backoff.go:259 +0x80
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/backoff.go:170
github.com/k8s-proxmox/cluster-api-provider-proxmox/cloud/scheduler.(*Scheduler).Run(0x400058c5a0)
	/workspace/cloud/scheduler/scheduler.go:174 +0xf0
created by github.com/k8s-proxmox/cluster-api-provider-proxmox/cloud/scheduler.(*Scheduler).RunAsync
	/workspace/cloud/scheduler/scheduler.go:184 +0x5c
I0522 15:22:10.633930       1 scheduler.go:268] "map[]" Name="qemu-scheduler" qemu="capi-mgmt-md-0-z8rlk-sjxvj"
I0522 15:22:10.634150       1 image.go:38] "setting cloud image" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/capi-mgmt-md-0-z8rlk-sjxvj" namespace="default" name="capi-mgmt-md-0-z8rlk-sjxvj" reconcileID=4604043f-8556-4577-8133-ddefbf46a400
panic: runtime error: integer divide by zero [recovered]
	panic: runtime error: integer divide by zero

goroutine 225 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x20?})
	/go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/runtime/runtime.go:56 +0xe0
panic({0x11f0140, 0x22faf10})
	/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/k8s-proxmox/cluster-api-provider-proxmox/cloud/scheduler/plugins/noderesource.(*NodeResource).Score(_, {_, _}, _, {0x0, {0x0, 0x0}, {0x13f660b, 0x9}, {0x0, ...}, ...}, ...)
	/workspace/cloud/scheduler/plugins/noderesource/node_resrouce.go:29 +0x64
github.com/k8s-proxmox/cluster-api-provider-proxmox/cloud/scheduler.(*Scheduler).RunScorePlugins(_, {_, _}, _, {0x0, {0x0, 0x0}, {0x13f660b, 0x9}, {0x0, ...}, ...}, ...)
	/workspace/cloud/scheduler/scheduler.go:353 +0x32c
github.com/k8s-proxmox/cluster-api-provider-proxmox/cloud/scheduler.(*Scheduler).SelectNode(_, {_, _}, {0x0, {0x0, 0x0}, {0x13f660b, 0x9}, {0x0, 0x0}, ...})
	/workspace/cloud/scheduler/scheduler.go:291 +0x19c
github.com/k8s-proxmox/cluster-api-provider-proxmox/cloud/scheduler.(*Scheduler).ScheduleOne(0x400058c5a0, {0x16724c8?, 0x40001fc4b0?})
	/workspace/cloud/scheduler/scheduler.go:210 +0x244
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
	/go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/backoff.go:259 +0x2c
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x40004dfe28?)
	/go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/backoff.go:226 +0x40
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x165c108, 0x4000b46180}, 0x1, 0x4000590540)
	/go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/backoff.go:227 +0x90
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x40001fc4b0?, 0x0, 0x0, 0x0?, 0x140a5a1?)
	/go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/backoff.go:204 +0x80
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x16724c8, 0x40001fc4b0}, 0x40004dff90, 0x140a5a1?, 0x17?, 0x0?)
	/go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/backoff.go:259 +0x80
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/backoff.go:170
github.com/k8s-proxmox/cluster-api-provider-proxmox/cloud/scheduler.(*Scheduler).Run(0x400058c5a0)
	/workspace/cloud/scheduler/scheduler.go:174 +0xf0
created by github.com/k8s-proxmox/cluster-api-provider-proxmox/cloud/scheduler.(*Scheduler).RunAsync
	/workspace/cloud/scheduler/scheduler.go:184 +0x5c

If i try to generate ticket manually using same credentials, all works fine:

❯ curl -k -d 'username=svc-pve-bt01@MYDOMAIN.NET' -d 'password=MyPa$$w0rd' https://pve.mydomain.net/api2/json/access/ticket | jq
{
  "data": {
    "ticket": "PVE:svc-pve-bt01@MYDOMAIN.NET:<REDACTED>",
    "cap": {
      "access": {},
      "mapping": {},
      "dc": {},
      "vms": {
        "VM.Backup": 1,
        "VM.Audit": 1,
        "VM.Allocate": 1,
        "VM.Config.HWType": 1,
        "VM.Snapshot.Rollback": 1,
        "VM.Config.Network": 1,
        "VM.Console": 1,
        "VM.Config.CDROM": 1,
        "VM.PowerMgmt": 1,
        "VM.Migrate": 1,
        "VM.Config.Cloudinit": 1,
        "VM.Config.Disk": 1,
        "VM.Config.Memory": 1,
        "VM.Clone": 1,
        "VM.Snapshot": 1,
        "VM.Config.Options": 1,
        "VM.Config.CPU": 1,
        "VM.Monitor": 1
      },
      "storage": {
        "Datastore.AllocateSpace": 1,
        "Datastore.Audit": 1
      },
      "sdn": {
        "SDN.Audit": 1,
        "SDN.Use": 1
      },
      "nodes": {}
    },
    "username": "svc-pve-bt01@MYDOMAIN.NET",
    "CSRFPreventionToken": "<REDACTED>"
  }
}

Environment:

• Cluster-api-provider-proxmox version: v0.3.5
• Proxmox VE version: 8.1.4
• Kubernetes version: (use kubectl version): v1.29.2

[justinas-b]

Is there a way to enable debug logs? i have tried adding -v=9, but it does not increase verbosity

[justinas-b]

I have a feeling that this provider works only when using user from PAM Realm, but i cannot find it mentioned explicitly anywhere. Could someone confirm this please?

[sp-yduck]

yes I think currently it's working with only PAM Realm.
If I remember/understand correctly this is because one of the endpoint of proxmox is available with only PAM Realm. (issue #65)

[justinas-b]

Hmm, interesting part is that it fails authentication for api endpoints that support it 👀