Optimize memory footprint of csi-addons

[black-dragon74]

With the option to specify precedence in the ConfigMap, we have an opportunity to optimize the memory footprint of csi-addons.

If the precedence is set to sc-only, there is no need to watch PVC resources. The current PVC controller handles most of the heavy lifting, but it can be refactored into two separate controllers.

One controller will watch StorageClass objects, and the other will watch PersistentVolumeClaim objects. Only one of these controllers needs to run, depending on the specified precedence.

We can further reduce memory usage by watching only PartialObjectMetadata instead of the full object.

To summarize:

• Split the controller into separate ones for StorageClass and PVC
• Run only one controller based on the configured precedence
• Watch only one resource Kind, not both
• Use PartialObjectMetadata to avoid caching full objects

[nixpanic]

Is there any loss in functionality when only one of the controllers will be enabled? What would be the advantage to enable one over an other? Is there a need to have both controllers at all?

Nit: precedence suggests to me that something needs to exist before something new, maybe an other name for the option would be clearer.

[Madhu-1]

Irrespective of the precedence a optimization is required for 2 reasons

• caching the whole PVC object
• Doing List operation which is expensive in lager clusters.