Create VolumeReplications from an annotation on the PVC

[SkalaNetworks]

In some instances, we may wish to create VolumeReplications for a lot of PVCs on the cluster.

When creating those PVCs by hand, it's easy to create the linked VolumeReplication, even if cumbersome.

But in some instances, the PVC is created by an operator that doesn't understand VolumeReplications. Creating the VolumeReplication for that PVC in an automated fashion would be hard without native support for it in the CSI-Addons.

Example:

• Have a Kubernetes cluster provisioned over Kubevirt VMs
• Have the Kubevirt CSI in that cluster (every PVC created in the tenant cluster is created on the host Kubernetes cluster)

If we wish to have every PVC in that cluster be automatically replicated, we need to manually create a VolumeReplication in the host cluster whenever someone provisions a new PVC in the tenant cluster. But if the PVCs are annotated by the CSI, we could have the CSI-Addons automatically provision the VolumeReplication.

[Madhu-1]

In some instances, we may wish to create VolumeReplications for a lot of PVCs on the cluster.

When creating those PVCs by hand, it's easy to create the linked VolumeReplication, even if cumbersome.

But in some instances, the PVC is created by an operator that doesn't understand VolumeReplications. Creating the VolumeReplication for that PVC in an automated fashion would be hard without native support for it in the CSI-Addons.

Example:

• Have a Kubernetes cluster provisioned over Kubevirt VMs
• Have the Kubevirt CSI in that cluster (every PVC created in the tenant cluster is created on the host Kubernetes cluster)

If we wish to have every PVC in that cluster be automatically replicated, we need to manually create a VolumeReplication in the host cluster whenever someone provisions a new PVC in the tenant cluster. But if the PVCs are annotated by the CSI, we could have the CSI-Addons automatically provision the VolumeReplication.

@SkalaNetworks annotation cannot be the source of truth for a new feature. Assume replication is just like snapshot its 1:1 mapping with the PVC. we dont want to build a controller that acts on the annotation where internally create the resource am pretty sure we will end up with even larger problems. The fact is that Replication is a constly operations and there are no direct ways to control it from the admin point, With RBAC and Replications CR the admin can control that who will be allowed to use the replication and what should be the replication interval etc.

[SkalaNetworks]

@Madhu-1 I get your point.

I think VolumeGroupReplications could fix my problem because they use labels to target PVCs, but my understanding is that they provide a crash consistant replication of every volume as if it was one unit. Which means I don't have the granularity of having a single VolumeReplication per PVC.

Another option would be to have a ressource just like VolumeReplicationGroups, but that creates a single VolumeReplication per PVC? It would use label selectors to generate them. A sort of VolumeReplicationSet akin to ReplicaSets

[SkalaNetworks]

@Madhu-1 coming back to this issue, as I'll be creating large amounts of PVCs that need to be replicated:

• Generating the VRs is cumbersome. I need to infer the VR class from the storage class of the PVC (which is not always doable, for example if doing GitOps, I don't expect to know that information all the time)
• SC to VRC isn't 1:1, for each storageclass, I might have 10 VRC, each with a different replication interval, but the same credentials. I need a way for the user to know which VRC to use. But what if there's no storageClass configured? Then, I need the user to pick the volumereplicationclass of the default storageclass that will be computed at runtime.

It can get quite hard to correctly infer the VRC. And I think automatically generating the VolumeReplication might be the best idea there. I know VRs can be expensive, but there's legitimate reasons to automatically create them.

I'm using VRs on Kubevirt VM disks, with the option given to the user to have disaster recovery of their VM.

I don't see many alternatives, unless I make my own controller, but I'm sure this feature could benefit the ecosystem if it was upstreamed.

[Madhu-1]

SC to VRC isn't 1:1, for each storageclass, I might have 10 VRC, each with a different replication interval, but the same credentials. I need a way for the user to know which VRC to use. But what if there's no storageClass configured? Then, I need the user to pick the volumereplicationclass of the default storageclass that will be computed at runtime.

yes thats true,VRC is just like SC, a single pool can be used by multiple SC. we can provide a feature to have default VRC as well (we need to check how backward compatible it is), Like the way the admin provides the SC name, admin need to provide the VRC name as well, the problem with going annotation is that it will become user operation not admin operation, the admin cannot control who uses the replication etc.

[SkalaNetworks]

@Madhu-1 I agree about your issue with VRs being "admin controlled"

Here's what I propose:

• Add an annotation (on a VRC) to make a VRC default for its provider (just like with SCs). This has the benefit of making VRs easier to create, and enforcing a default replication period if none is provided. This has no impact on the admin problem, it just means a volume replication can be created with no volumeReplicationClass specified (this behaviour exists for storage classes and volume snapshot classes)
• Add a new option (disabled by default) to watch PVCs and lookup an annotation on the PVC to auto-generate a VolumeReplication. The annotation would contain the name of the VolumeReplicationClass to use. When the controller sees that, it generates a matching VolumeReplication with the given VRC.

The last option, being disabled by default, doesn't change the normal course of things. But if someone has a cluster where PVCs are not generated by developpers/users without any filtering, they can enable it.

This feature can be especially useful when PVCs are generated by third-party controllers.

Say you're deploying a Kubernetes cluster using Kubevirt+Cluster API, the PVCs will be created by the Cluster API through Kubevirt.

There's no easy way to automatically enroll these PVCs in a replication, as I'd need to programatically lookup all the work that was done by Kubevirt and then, for each PVC, generate a corresponding VolumeReplication. But if I set an annotation on the volume of my Kubevirt VM, the PVC will inherit it, and the corresponding VR will be automatically generated.

WDYT

[Madhu-1]

I had the same thought in mind i didnt write it earlier as i still do have some concerns with automating at the PVC level.

what about the additional field need to be set on the VR object apart from PVC i dont want to build everything from the PVC annotation as it will get in to same complexity?

what about the plan to automation of failover and failback and it need to be done at the VR level.

[SkalaNetworks]

what about the additional field need to be set on the VR object apart from PVC i dont want to build everything from the PVC annotation as it will get in to same complexity?

Do you mean you want an annotation on the VR and not on the PVC? I'm not sure I see how to build from something else than the PVC, is there another way you have in mind?

For the failover/fallback, you have the VolumeReplications generated for each PVC (assuming we use that annotation on a PVC), so you could just edit them to set them to primary/secondary/resync

That wouldn't be the hardest part, as it usually requires a manual operation from an admin to reconcile the clusters.