Race condition with overwrite=true on proxmox_virtual_environment_file -> Error: failed to read file from

[MacherelR]

Summary

I’m seeing intermittent errors with proxmox_virtual_environment_file when using overwrite = true for ISO uploads.

In some runs, the resource logs:

Warning: the existing file "secondary_01:iso/<name>_seed.iso" has been overwritten by the resource
Error: failed to read file from "secondary_01:iso/<name>_seed.iso"

The ISO file does exist in Proxmox afterwards, so it looks like the upload actually succeeds but the provider fails when reading back metadata. This feels like a timing / race condition between the upload and the follow-up read.
I feel like this happens because of parallelism as I create multiple ISOs and for the one getting the error I do not have the classic
module.cloud_init["service"].proxmox_virtual_environment_file.seed_iso: Creation complete after Xs message.

Basically an example of logs I get would be:

module.cloud_init["ex-01-svc-2"].proxmox_virtual_environment_file.seed_iso: Creating...
module.cloud_init["ex-01-svc-3"].proxmox_virtual_environment_file.seed_iso: Creating...
module.cloud_init["ex-01-svc-4"].proxmox_virtual_environment_file.seed_iso: Creating...
module.cloud_init["ex-01-svc-1"].proxmox_virtual_environment_file.seed_iso: Creating...
module.cloud_init["ex-01-svc-0"].proxmox_virtual_environment_file.seed_iso: Creating...
module.cloud_init["ex-01-svc-2"].proxmox_virtual_environment_file.seed_iso: Creation complete after 3s [id=secondary_01:iso/ex-01-svc-2_a6c50c31a1fa1bdb915999bca57fa8c1_seed.iso]
module.cloud_init["ex-01-svc-1"].proxmox_virtual_environment_file.seed_iso: Creation complete after 5s [id=secondary_01:iso/ex-01-svc-1_ff87053e8780489c95c9a793fcd16b52_seed.iso]
module.cloud_init["ex-01-svc-0"].proxmox_virtual_environment_file.seed_iso: Creation complete after 5s [id=secondary_01:iso/ex-01-svc-0_63195b7bc9c641aa6b700928a2a9655d_seed.iso]
module.cloud_init["ex-01-svc-4"].proxmox_virtual_environment_file.seed_iso: Creation complete after 5s [id=secondary_01:iso/ex-01-svc-4_86ade32ca8fff9f782e0250e8e15299c_seed.iso]
╷
│ Error: failed to read file from "secondary_01:iso/ex-01-svc-3_eb996d813bfdf3150a0f3b20d570b0bf_seed.iso"
│ 
│   with module.cloud_init["ex-01-svc-3"].proxmox_virtual_environment_file.seed_iso,
│   on ../modules/cloud_init/cloud_init.tf line 87, in resource "proxmox_virtual_environment_file" "seed_iso":
│   87: resource "proxmox_virtual_environment_file" "seed_iso" {
│ 
╵

With -parallelism=1, the problem is much harder to reproduce (I haven’t hit the error again yet), but I still see the overwrite warning, so the same code path is in use.

I’d like to know if this is a known limitation, or if there’s a recommended pattern for this use case (cloud-init ISO with overwrite = true).

Environment

CLI: OpenTofu (via atmos terraform deploy)

Atmos: atmos terraform deploy -s

Terraform/OpenTofu flags: sometimes -parallelism=10 (default), sometimes -parallelism=1

Use case: generate cloud-init seed ISOs locally (user-data / meta-data / network-config via genisoimage), then upload them to Proxmox and attach to VMs to avoid using ssh for cloud-init.

[bpg]

Are you writing are file to a shared storage? The same file simultaneously from multiple nodes?
Are you sure it is a valid use case?

[MacherelR]

It is indeed a shared storage through 3 nodes (ceph) but at the moment of the error, the terraform is the only user to use it.