Where is decompression for the resource proxmox_virtual_environment_download_file done?

[Tanchwa]

Hello all,

I'd like to work on a PR to support other decompression algos because one of the images I need to pull is an xz file, but not a tarball. So it's not quite behaving properly with the ones available.

I found where it does the validation in the code, but I can't seem to find where the actual decompression is being done. Is that done on proxmox side automatically for some file types? If so, if I WERE to add a helper for other decompression algos, where should that go?

Thanks,
Tanchwa

[bpg]

Hi @Tanchwa! 👋🏼

The proxmox_virtual_environment_download_file resource is a wrapper around the download-url PVE API. The download and decompression happens on PVE, the provider just submits a job (you can see it in list of jobs in PVE UI), and PVE node does all the work. The file content does not leave PVE.

The situation is different for proxmox_virtual_environment_file resource. It does a "custom" upload over SSH for snippets, but uses upload PVE API for uploading large files, and they must be of the format supported by PVE depends of the storage type (ref). The differences between these two resources is that the proxmox_virtual_environment_file buffers everything locally in a tmp file first, then uploads the file over API. Which is super inefficient, but might serve your purpose.
In theory, you can download any file format locally, then uncompress (or otherwise transcode it as needed), then upload. Which will be a huge waste of disk space, unless you do a stream decompression/transcoding/multipart uploading on the fly. I think it is doable.

But regardless of the technicalities, you'd need to come up with some metadata model that would define a pipeline to convert a source file to one of the PVE-supported formats. Particularly for your use case of "an xz file, but not a tarball" how to convert the "not a tarball" format to a tarball before compressing it with xz or gz.

[Tanchwa]

right now, I use the download file resource to download the file and stick in a filename unxz can use, then I use the remote exec provissioner to decompress the file. I finally stand up the virtual machine by passing in the path to the ISO, but I think I've found a different bug. It seems the VM provissioner resource tries to name the resulting LVM created based on the entire path provided, and not just the image name. I will open an issue when I have a second.

  disk {
    datastore_id = "local-lvm"
    //file_id      = proxmox_virtual_environment_download_file.home_assistant_img.id
    path_in_datastore = "/var/lib/vz/template/iso/haos_generic-x86-12.2.img"
    interface         = "scsi0"
  }

│ Error: error creating VM: received an HTTP 500 response - Reason: lvm name '/var/lib/vz/template/iso/haos_generic-x86-12.2.img' contains illegal characters
│
│   with module.home_assistant.proxmox_virtual_environment_vm.home_assistant_vm,
│   on Modules/Home-Assistant/main.tf line 1, in resource "proxmox_virtual_environment_vm" "home_assistant_vm":
│    1: resource "proxmox_virtual_environment_vm" "home_assistant_vm" {
│
╵

[bpg]

path_in_datastore is for a attaching disks from a different VMs, not for creating new disks from images.

Have you checked https://registry.terraform.io/providers/bpg/proxmox/latest/docs/guides/cloud-image#create-a-vm-from-an-exiting-image-on-proxmox, would it help in your use case?

[Tanchwa]

hardcoding that new file path as the ID worked, thanks. Now I'm stuck figuring out how to debug the image.

[maxo99]

If any others find this thread, in my case trying to get homeassistant os to decompress from .xz,

Check out this discussion: (#1212)
TLDR: zst will decompress the .xz

Which did work for me using:

resource "proxmox_virtual_environment_download_file" "haos_image" {
  content_type           = "iso"
  datastore_id            = "local"
  node_name             = "pve-01"
  url                             = "https://github.com/home-assistant/operating-system/releases/download/16.2/haos_ova-16.2.qcow2.xz"
  file_name                = "haos_ova-16.2.qcow2.xz.img"
  decompression_algorithm = "zst"
}

[MisterMountain]

Yes works for me, thank you so much for the heads up!

[maxo99]

Good to hear.
If no other issues with the .qcow2.xz image install please look to close #2208
Thanks for confirming resolution