Tinkerbell CAPT, Talos CABPT & CACPPT

[frederic-loui]

Hi !

First of all, I would like to thank all the Tinkerbell crew for this awesome platform. We undertook extensive tests with Tinkerbell stack and CAPT and all is working flawlessly. However, we did try to deploy Talos machine using Talos CABPT & CACPPT combined with Tinkerbell CAPT:

1- A dynamic Workflow is created
2- The dynamically created template is successfully rendered and has a templateRef in -1-
3- Hardware trigger in hardwareMap has a hardwareRef also is -1-
4- We used templateOverride in order to stream Talos raw image after the stream the machine reboots

But after that point, the Talos stays is in maintenance mode.

At that point, in order to complete the operation one should issue:

talosctl apply config <endpoint> <node> <talos-node-config>
talos bootstrap  <endpoint> <node> <talosconfg>

Considering Talos installation workflow, do you see inherent "obvious" reason that Tinkerbell CAPT would not be compatible with Talos CABPT/CACPPT ? If yes, in that case we will keep Tinkerbell for BARE METAL workload and select an other one compatible with TALOS CABPT/CACPPT.

Thanks for your help !

PS: If needed i can provide more artefacts, but before I wanted to ensure if there is (or not) inherent incompatibility between Tinkerbell CAPT infrastructure provider and Talos bootstrap/controlplane provider.

[jacobweinstock]

Hey @frederic-loui . Thank you for posting and for using Tinkerbell and CAPT! I'm not familiar with the Talos Linux CAPI providers. In terms of CAPT, one option to get this working might be the following:

In your templateOverride Template, use a 2nd task that runs on another Tink Worker. This Tink Worker would need to be a running Worker somewhere that has network access to all the machines that will get provisioned. The second Task would be responsible for 1. waiting until the Talos endpoint is up and available. 2. Run the 2 commands you posted above. Here's a contrived example:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: TinkerbellMachineTemplate
metadata:
  name: example
spec:
  template:
    spec:
      templateOverride: |-
        version: "0.1"
        name: example-template
        global_timeout: 6000
        tasks:
          - name: "stream Talos"
            worker: "{{.device_1}}"
            volumes:
              - /dev:/dev
            actions:
              - name: "stream image"
                image: quay.io/tinkerbell/actions/image2disk
                timeout: 1200
                environment:
                  IMG_URL: http://you-server/talos.gz
                  DEST_DISK: {{ index .Hardware.Disks 0 }}
                  COMPRESSED: true
          - name: "Talos operations"
            worker: "talos-worker"
            actions:
              - name: "Wait for endpoint"
                image: alpine
                command: [
                  "sh", "-c",
                  "until ping -c1 -W1 ${TALOS_ENDPOINT} >/dev/null 2>&1; do sleep 1; done", 
                ]
                timeout: 1200
                volumes:
                  - /talos:/talos
                environment:
                  TALOS_ENDPOINT: {{ (index .Hardware.Interfaces 0).DHCP.IP.Address }}
              - name: "Talos apply"
                image: ghcr.io/siderolabs/talosctl
                command: [
                  "sh", "-c",
                  "talosctl apply config ${TALOS_ENDPOINT} ${TALOS_NODE} ${TALOS_CONFIG}", 
                ]
                timeout: 90
                volumes:
                  - /talos:/talos
                environment:
                  TALOS_ENDPOINT: {{ (index .Hardware.Interfaces 0).DHCP.IP.Address }}
                  TALOS_NODE: {{ (index .Hardware.Interfaces 0).DHCP.IP.Hostname }}
                  TALOS_CONFIG: /talos/config.file
              - name: "Talos bootstrap"
                image: ghcr.io/siderolabs/talos
                command: [
                  "sh", "-c",
                  "talos bootstrap ${TALOS_ENDPOINT} ${TALOS_NODE} ${TALOS_CONFIG}", 
                ]
                timeout: 90
                volumes:
                  - /talos:/talos
                environment:
                  TALOS_ENDPOINT: {{ (index .Hardware.Interfaces 0).DHCP.IP.Address }}
                  TALOS_NODE: {{ (index .Hardware.Interfaces 0).DHCP.IP.Hostname }}
                  TALOS_CONFIG: /talos/config.file

For this to work, you'll need to have a long-running Tink Worker up and running has an ID equal to talos-worker and that has access to the nodes and the Talos configs.

Hope this makes sense. Happy to clarify or talk through this more. Thanks!

[frederic-loui]

Hi @jacobweinstock ! Many thanks for your swift answer ! What you proposed makes sense but would not work due to Talos installation mode of operation. Let me expose what i would call "Talos manual installation mode of operation"

PHASE 1:
1- BARE METAL SERVER is empy and boot (iso or pxe) in this example let's consider iso
2- Once booted Talos installer is loaded into BMS RAM
3- TALOS gets an IP from DHCP and is reachable from IPv4 or IPv6 point of view
4- At that point TALOS is in RAM and is in maintenance mode (no installation is done so far)

PHASE 2:
1-From a client host where talosctl is installed that has connectivity to the BMS above
2- We generate the config. talos generate config .... This process generate Talos k8s control-plane and worker configuration files
3- The we apply the config to the first control-plane node. talosctl apply config control-plane.yaml
4- At that point, an effective installation process is triggered
5- the machine then reboot

PHASE 3:
1- After reboot node begins installation process on the disk (/dev/sda for example)
2- if this is the first control-plane node, we need to issue talosctl bootstrap command which following Talos terminology attach etcd cluster to the first new ly created control-plane node.
3- Default CNI is installed and your cluster becomes ready

When using SIDERO METAL provider infrastructure all PHASE 1,2,3 are subsequently trigged using the three CAPI components:
1- Bootstrap component: Talos Cluster API Bootstrap Provider Talos (CABPT)
2- Control-plane component: Talos Cluster API Control-Plane Provider Talos (CACPPT)
3- Infrastructure component: SIDERO METAL

And we end up in getting a full cluster deployed with clusterctl using 1/2/3 above.

As was assuming that -3- infrastructure provider was seamlessly interchangeable and decided to use -3- Tinkerbell CAPT as infrastructure provider. We managed somehow to trigger a dynamic workflow + template but we did not managed to get hook working as expected in PHASE-1 (load talos installer in RAM), PHASE-2 (Apply config from external host while the talos node is running in maintenance mode),PHASE-3 (have node installed with applied config in -2-).

Out of curiosity did you had request from customer regarding TALOS deployment using Tinkerbell CAPT ?

Other than that we use Tinkerbell for machine deployment or pure K8s deployment and all is working flawlessly.
(except that CAPT does not work for K8s cluster 1.33.x)

PS: Just in case it can help this is a blog article with TALOS CABPT+CAPPT+Promox Infrastructure Provider. You'll wind a video describing the talos installation process.

[mcanevet]

@frederic-loui I'm trying to do the same thing. Have you found a solution? Would you mind sharing your manifests?

[frederic-loui]

Hi, we were unable to have CAPI=CABPT+CACPPT+TINKERBELL CAPT.
Hence in our context, we decided to keep TINKERBELL for BAREMETAL and regarding TALOS deployment we just used SIDERO METAL CAPI This choice is not perfect because the solution is deprecated in favor of OMNI. but it has the strength to fit exactly our needs and constraint.

However, if at some point CAPI specification becomes more modular in term of interdependency, i would still be willing to use TINKERBELL CAPT as a more generic solution. (for TALOS, BMS or any OS deployment)

[mcanevet]

@frederic-loui would you mind sharing your yaml manifests of the time you where trying with CABPT+CACPPT+CAPT if you still have it?

[mcanevet]

I could initiate the cluster bootstrap with this, but somehow it fails:

---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: TinkerbellCluster
metadata:
  name: capt-playground
  namespace: tinkerbell
spec:
  controlPlaneEndpoint:
    host: 172.18.10.21
    port: 6443
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: capt-playground
  namespace: tinkerbell
spec:
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
    kind: TalosControlPlane
    name: capt-playground-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: TinkerbellCluster
    name: capt-playground
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: TalosControlPlane
metadata:
  name: capt-playground-control-plane
  namespace: tinkerbell
spec:
  version: v1.34.0
  replicas: 1
  infrastructureTemplate:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: TinkerbellMachineTemplate
    name: capt-playground-control-plane
    namespace: tinkerbell
  controlPlaneConfig:
    controlplane:
      generateType: controlplane
      strategicPatches:
        - |
          - op: replace
            path: /machine/install
            value:
              disk: /dev/vda
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: TinkerbellMachineTemplate
metadata:
  name: capt-playground-control-plane
  namespace: tinkerbell
spec:
  template:
    spec:
      bootOptions:
        bootMode: netboot
      hardwareAffinity:
        required:
        - labelSelector:
            matchLabels:
              tinkerbell.org/role: control-plane
      templateOverride: |-
        version: "0.1"
        name: talos-install-template
        global_timeout: 1800
        tasks:
          - name: "install-talos"
            worker: "{{.device_1}}"
            volumes:
              - /dev:/dev
              - /dev/console:/dev/console
              - /lib/firmware:/lib/firmware:ro
            actions:
              - name: "stream talos image"
                image: quay.io/tinkerbell/actions/image2disk
                timeout: 1200
                environment:
                  IMG_URL: https://github.com/siderolabs/talos/releases/download/v1.11.1/metal-amd64.raw.zst
                  DEST_DISK: {{ index .Hardware.Disks 0 }}
                  COMPRESSED: true
              - name: "kexec"
                image: ghcr.io/jacobweinstock/waitdaemon:latest
                timeout: 90
                pid: host
                environment:
                  BLOCK_DEVICE: {{ formatPartition ( index .Hardware.Disks 0 ) 3 }}
                  FS_TYPE: xfs
                  KERNEL_PATH: /A/vmlinuz
                  INITRD_PATH: /A/initramfs.xz
                  CMD_LINE: "talos.platform=metal talos.config=http://172.18.10.74:7172/2009-04-04/user-data console=tty0 init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 printk.devkmsg=on selinux=1 net.ifnames=0"
                  IMAGE: quay.io/tinkerbell-actions/kexec:v1.0.0
                  WAIT_SECONDS: 10
                volumes:
                  - /var/run/docker.sock:/var/run/docker.sock

[mcanevet]

@frederic-loui FYI I could make it work with this in case you're still interested:

---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: TinkerbellCluster
metadata:
  name: capt-playground
  namespace: tinkerbell
spec:
  controlPlaneEndpoint:
    host: 172.18.10.21
    port: 6443
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: capt-playground
  namespace: tinkerbell
spec:
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
    kind: TalosControlPlane
    name: capt-playground-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: TinkerbellCluster
    name: capt-playground
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: TalosControlPlane
metadata:
  name: capt-playground-control-plane
  namespace: tinkerbell
spec:
  version: v1.34.0
  replicas: 1
  infrastructureTemplate:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: TinkerbellMachineTemplate
    name: capt-playground-control-plane
    namespace: tinkerbell
  controlPlaneConfig:
    controlplane:
      generateType: controlplane
      strategicPatches:
        - |
          - op: replace
            path: /machine/install
            value:
              disk: /dev/vda
              image: ghcr.io/siderolabs/installer:v1.11.1
              wipe: false
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: TinkerbellMachineTemplate
metadata:
  name: capt-playground-control-plane
  namespace: tinkerbell
spec:
  template:
    spec:
      bootOptions:
        bootMode: netboot
      hardwareAffinity:
        required:
        - labelSelector:
            matchLabels:
              tinkerbell.org/role: control-plane
      templateOverride: |-
        version: "0.1"
        name: talos-install-template
        global_timeout: 1800
        tasks:
          - name: "install-talos"
            worker: "{{.device_1}}"
            volumes:
              - /dev:/dev
              - /dev/console:/dev/console
              - /lib/firmware:/lib/firmware:ro
            actions:
              - name: "stream talos image"
                image: quay.io/tinkerbell/actions/image2disk:latest
                timeout: 1200
                environment:
                  DEST_DISK: {{ index .Hardware.Disks 0 }}
                  IMG_URL: https://github.com/siderolabs/talos/releases/download/v1.11.1/metal-amd64.raw.zst
                  COMPRESSED: true
              - name: "kexec"
                image: ghcr.io/jacobweinstock/waitdaemon:latest
                timeout: 90
                pid: host
                environment:
                  BLOCK_DEVICE: {{ formatPartition ( index .Hardware.Disks 0 ) 3 }}
                  FS_TYPE: xfs
                  KERNEL_PATH: /A/vmlinuz
                  INITRD_PATH: /A/initramfs.xz
                  IMAGE: quay.io/tinkerbell-actions/kexec:v1.0.0
                  CMD_LINE: "talos.platform=metal talos.config=http://172.18.10.74:7172/2009-04-04/user-data console=tty0 init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 printk.devkmsg=on selinux=1"
                  WAIT_SECONDS: 10
                volumes:
                  - /var/run/docker.sock:/var/run/docker.sock

---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: capt-playground-worker-a
  namespace: tinkerbell
spec:
  clusterName: capt-playground
  replicas: 1
  selector:
    matchLabels: null
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
          kind: TalosConfigTemplate
          name: talosconfig-workers
      clusterName: capt-playground
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: TinkerbellMachineTemplate
        name: capt-playground-worker-a
      version: v1.34.0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: TinkerbellMachineTemplate
metadata:
  name: capt-playground-worker-a
  namespace: tinkerbell
spec:
  template:
    spec:
      bootOptions:
        bootMode: netboot
      hardwareAffinity:
        required:
        - labelSelector:
            matchLabels:
              tinkerbell.org/role: worker
      templateOverride: |-
        version: "0.1"
        name: install-talos
        global_timeout: 6000
        tasks:
          - name: "playground-template"
            worker: "{{.device_1}}"
            volumes:
              - /dev:/dev
              - /dev/console:/dev/console
              - /lib/firmware:/lib/firmware:ro
            actions:
              - name: "stream talos image"
                image: quay.io/tinkerbell/actions/image2disk:latest
                timeout: 600
                environment:
                  DEST_DISK: {{ index .Hardware.Disks 0 }}
                  IMG_URL: "https://github.com/siderolabs/talos/releases/download/v1.11.1/metal-amd64.raw.zst"
                  COMPRESSED: true
              - name: "kexec"
                image: ghcr.io/jacobweinstock/waitdaemon:latest
                timeout: 90
                pid: host
                environment:
                  BLOCK_DEVICE: {{ formatPartition ( index .Hardware.Disks 0 ) 3 }}
                  FS_TYPE: xfs
                  KERNEL_PATH: /A/vmlinuz
                  INITRD_PATH: /A/initramfs.xz
                  IMAGE: quay.io/tinkerbell-actions/kexec:v1.0.0
                  CMD_LINE: "talos.platform=metal talos.config=http://172.18.10.74:7172/2009-04-04/user-data console=tty0 init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 printk.devkmsg=on selinux=1"
                  WAIT_SECONDS: 10
                volumes:
                  - /var/run/docker.sock:/var/run/docker.sock
---
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: TalosConfigTemplate
metadata:
  name: talosconfig-workers
  namespace: tinkerbell
spec:
  template:
    spec:
      generateType: worker
      talosVersion: v1.11.1
      configPatches:
          - op: replace
            path: /machine/install
            value:
              disk: /dev/vda
              image: ghcr.io/siderolabs/installer:v1.11.1
              wipe: false

[frederic-loui]

Hi !

This is super interesting. TBH, as we had time constraint we simply moved to SIDERO METAL and since then it worked as expected.
Now your finding and immense effort is very interesting as we might get an alternative if one day SIDERO METAL is completely abandoned. My manifest did not work at all, and I was unable to even had TALOS bootstrap kicked in ....

We will test your manifest and get back to you for sure.

Thanks again !