Skip to content

Commit 157099e

Browse files
author
Arvind Thirumurugan
committed
add ClusterStagedWorkloadTracker and StagedWorkloadTracker
Signed-off-by: Arvind Thirumurugan <arvindth@microsoft.com>
1 parent e4f8936 commit 157099e

File tree

12 files changed

+323
-89
lines changed

12 files changed

+323
-89
lines changed

approval-controller-metric-collector/README.md

Lines changed: 32 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,15 @@ This solution introduces three new CRDs that work together with KubeFleet's nati
2929
- Contains collected `workload_health` metrics for all workloads in a cluster
3030
- Updated every 30 seconds by the metric collector
3131

32-
3. **WorkloadTracker** (cluster-scoped)
33-
- Defines which workloads to monitor and their health thresholds
32+
3. **ClusterStagedWorkloadTracker** (cluster-scoped)
33+
- Defines which workloads to monitor for a ClusterStagedUpdateRun
34+
- The name must match the ClusterStagedUpdateRun name
35+
- Specifies namespace, workload name, and expected health status
36+
- Used by approval-request-controller to determine if stage is ready for approval
37+
38+
4. **StagedWorkloadTracker** (namespaced)
39+
- Defines which workloads to monitor for a StagedUpdateRun
40+
- The name and namespace must match the StagedUpdateRun name and namespace
3441
- Specifies namespace, workload name, and expected health status
3542
- Used by approval-request-controller to determine if stage is ready for approval
3643

@@ -61,7 +68,9 @@ This solution introduces three new CRDs that work together with KubeFleet's nati
6168
4. **Health Evaluation**
6269
- Approval-request-controller monitors `MetricCollectorReports` from all stage clusters
6370
- Every 15 seconds, it:
64-
- Fetches the `WorkloadTracker` to know which workloads to check
71+
- Fetches the appropriate workload tracker:
72+
- For cluster-scoped: `ClusterStagedWorkloadTracker` with same name as ClusterStagedUpdateRun
73+
- For namespace-scoped: `StagedWorkloadTracker` with same name and namespace as StagedUpdateRun
6574
- For each cluster in the stage:
6675
- Reads its `MetricCollectorReport` from `fleet-member-<cluster-name>` namespace
6776
- Verifies all tracked workloads are present and healthy
@@ -171,18 +180,30 @@ The script performs the following:
171180
1. Builds the `approval-request-controller:latest` image
172181
2. Loads the image into the kind hub cluster
173182
3. Verifies that required kubefleet CRDs are installed
174-
4. Installs the controller via Helm with the custom CRDs (MetricCollector, MetricCollectorReport, WorkloadTracker)
183+
4. Installs the controller via Helm with the custom CRDs (MetricCollector, MetricCollectorReport, ClusterStagedWorkloadTracker, StagedWorkloadTracker)
175184
5. Verifies the installation
176185

177186
### 6. Configure Workload Tracker
178187

179-
Apply the WorkloadTracker to define which workloads to monitor:
188+
Apply the appropriate workload tracker based on which type of staged update you'll use:
189+
190+
#### For Cluster-Scoped Updates (ClusterStagedUpdateRun):
191+
192+
```bash
193+
# Apply ClusterStagedWorkloadTracker
194+
# Important: The name must match your ClusterStagedUpdateRun name
195+
kubectl apply -f ./examples/workloadtracker/clusterstagedworkloadtracker.yaml
196+
```
197+
198+
#### For Namespace-Scoped Updates (StagedUpdateRun):
180199

181200
```bash
182-
kubectl apply -f ./examples/workloadtracker/
201+
# Apply StagedWorkloadTracker
202+
# Important: The name and namespace must match your StagedUpdateRun name and namespace
203+
kubectl apply -f ./examples/workloadtracker/stagedworkloadtracker.yaml
183204
```
184205

185-
This tells the approval controller which workloads to track and what health thresholds to use.
206+
This tells the approval controller which workloads to track.
186207

187208
### 7. Install Metric Collector (Member Clusters)
188209

@@ -442,7 +463,10 @@ kubectl get metriccollectorreport -A
442463
- Ensure workloads have Prometheus scrape annotations
443464

444465
### Approvals not happening
445-
- Check WorkloadTracker resources define correct health thresholds
466+
- Check that the workload tracker name matches the update run name:
467+
- For ClusterStagedUpdateRun: ClusterStagedWorkloadTracker name must match
468+
- For StagedUpdateRun: StagedWorkloadTracker name and namespace must match
469+
- Verify workload tracker resources define correct health thresholds
446470
- Verify MetricCollectorReports are being created on the hub
447471
- Review approval-request-controller logs for decision-making details
448472

approval-controller-metric-collector/approval-request-controller/apis/metric/v1alpha1/workloadtracker_types.go

Lines changed: 43 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,11 @@ type WorkloadReference struct {
3838
// +kubebuilder:storageversion
3939
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
4040

41-
// WorkloadTracker expresses user intent to track certain workloads
42-
type WorkloadTracker struct {
41+
// ClusterStagedWorkloadTracker expresses user intent to track certain workloads for a ClusterStagedUpdateRun.
42+
// The name of this resource should match the name of the ClusterStagedUpdateRun it is used for.
43+
// For example, if the ClusterStagedUpdateRun is named "example-cluster-staged-run", the
44+
// ClusterStagedWorkloadTracker should also be named "example-cluster-staged-run".
45+
type ClusterStagedWorkloadTracker struct {
4346
metav1.TypeMeta `json:",inline"`
4447
metav1.ObjectMeta `json:"metadata,omitempty"`
4548

@@ -51,13 +54,47 @@ type WorkloadTracker struct {
5154
// +kubebuilder:object:root=true
5255
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
5356

54-
// WorkloadTrackerList contains a list of WorkloadTracker
55-
type WorkloadTrackerList struct {
57+
// ClusterStagedWorkloadTrackerList contains a list of ClusterStagedWorkloadTracker
58+
type ClusterStagedWorkloadTrackerList struct {
5659
metav1.TypeMeta `json:",inline"`
5760
metav1.ListMeta `json:"metadata,omitempty"`
58-
Items []WorkloadTracker `json:"items"`
61+
Items []ClusterStagedWorkloadTracker `json:"items"`
62+
}
63+
64+
// +genclient
65+
// +kubebuilder:object:root=true
66+
// +kubebuilder:resource:scope="Namespaced",categories={fleet,fleet-placement}
67+
// +kubebuilder:storageversion
68+
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
69+
70+
// StagedWorkloadTracker expresses user intent to track certain workloads for a StagedUpdateRun.
71+
// The name and namespace of this resource should match the name and namespace of the StagedUpdateRun it is used for.
72+
// For example, if the StagedUpdateRun is named "example-staged-run" in namespace "test-ns", the
73+
// StagedWorkloadTracker should also be named "example-staged-run" in namespace "test-ns".
74+
type StagedWorkloadTracker struct {
75+
metav1.TypeMeta `json:",inline"`
76+
metav1.ObjectMeta `json:"metadata,omitempty"`
77+
78+
// Workloads is a list of workloads to track
79+
// +optional
80+
Workloads []WorkloadReference `json:"workloads,omitempty"`
81+
}
82+
83+
// +kubebuilder:object:root=true
84+
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
85+
86+
// StagedWorkloadTrackerList contains a list of StagedWorkloadTracker
87+
type StagedWorkloadTrackerList struct {
88+
metav1.TypeMeta `json:",inline"`
89+
metav1.ListMeta `json:"metadata,omitempty"`
90+
Items []StagedWorkloadTracker `json:"items"`
5991
}
6092

6193
func init() {
62-
SchemeBuilder.Register(&WorkloadTracker{}, &WorkloadTrackerList{})
94+
SchemeBuilder.Register(
95+
&ClusterStagedWorkloadTracker{},
96+
&ClusterStagedWorkloadTrackerList{},
97+
&StagedWorkloadTracker{},
98+
&StagedWorkloadTrackerList{},
99+
)
63100
}

approval-controller-metric-collector/approval-request-controller/apis/metric/v1alpha1/zz_generated.deepcopy.go

Lines changed: 103 additions & 41 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../../../config/crd/bases/metric.kubernetes-fleet.io_clusterstagedworkloadtrackers.yaml
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../../../config/crd/bases/metric.kubernetes-fleet.io_stagedworkloadtrackers.yaml

approval-controller-metric-collector/approval-request-controller/charts/approval-request-controller/templates/crds/metric.kubernetes-fleet.io_workloadtrackers.yaml

Lines changed: 0 additions & 1 deletion
This file was deleted.

approval-controller-metric-collector/approval-request-controller/config/crd/bases/metric.kubernetes-fleet.io_workloadtrackers.yaml renamed to approval-controller-metric-collector/approval-request-controller/config/crd/bases/metric.kubernetes-fleet.io_clusterstagedworkloadtrackers.yaml

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,23 +4,27 @@ kind: CustomResourceDefinition
44
metadata:
55
annotations:
66
controller-gen.kubebuilder.io/version: v0.16.0
7-
name: workloadtrackers.metric.kubernetes-fleet.io
7+
name: clusterstagedworkloadtrackers.metric.kubernetes-fleet.io
88
spec:
99
group: metric.kubernetes-fleet.io
1010
names:
1111
categories:
1212
- fleet
1313
- fleet-placement
14-
kind: WorkloadTracker
15-
listKind: WorkloadTrackerList
16-
plural: workloadtrackers
17-
singular: workloadtracker
14+
kind: ClusterStagedWorkloadTracker
15+
listKind: ClusterStagedWorkloadTrackerList
16+
plural: clusterstagedworkloadtrackers
17+
singular: clusterstagedworkloadtracker
1818
scope: Cluster
1919
versions:
2020
- name: v1alpha1
2121
schema:
2222
openAPIV3Schema:
23-
description: WorkloadTracker expresses user intent to track certain workloads
23+
description: |-
24+
ClusterStagedWorkloadTracker expresses user intent to track certain workloads for a ClusterStagedUpdateRun.
25+
The name of this resource should match the name of the ClusterStagedUpdateRun it is used for.
26+
For example, if the ClusterStagedUpdateRun is named "example-cluster-staged-run", the
27+
ClusterStagedWorkloadTracker should also be named "example-cluster-staged-run".
2428
properties:
2529
apiVersion:
2630
description: |-

0 commit comments

Comments
 (0)