improve readme

Signed-off-by: Arvind Thirumurugan <arvindth@microsoft.com>

Author: Arvind Thirumurugan

approval-controller-metric-collector/README.md

@@ -10,6 +10,93 @@ This directory contains two controllers:
 
 ![Approval Controller and Metric Collector Architecture](./approval-controller-metric-collector.drawio.png)
 
+## How It Works
+
+### Custom Resource Definitions (CRDs)
+
+This solution introduces three new CRDs that work together with KubeFleet's native resources:
+
+#### Hub Cluster CRDs
+
+1. **MetricCollector** (cluster-scoped)
+   - Defines Prometheus connection details and where to report metrics
+   - Gets propagated to member clusters via ClusterResourcePlacement (CRP)
+   - Each member cluster receives a customized version with its specific `reportNamespace`
+
+2. **MetricCollectorReport** (namespaced)
+   - Created by metric-collector on member clusters, reported back to hub
+   - Lives in `fleet-member-<cluster-name>` namespaces on the hub
+   - Contains collected `workload_health` metrics for all workloads in a cluster
+   - Updated every 30 seconds by the metric collector
+
+3. **WorkloadTracker** (cluster-scoped)
+   - Defines which workloads to monitor and their health thresholds
+   - Specifies namespace, workload name, and expected health status
+   - Used by approval-request-controller to determine if stage is ready for approval
+
+### Automated Approval Flow
+
+1. **Stage Initialization**
+   - User creates an UpdateRun (`ClusterStagedUpdateRun` or `StagedUpdateRun`) on the hub
+   - KubeFleet creates an ApprovalRequest (`ClusterApprovalRequest` or `ApprovalRequest`) for the first stage
+   - The ApprovalRequest enters "Pending" state, waiting for approval
+
+2. **Metric Collector Deployment**
+   - Approval-request-controller watches the CAR
+   - Creates a `MetricCollector` resource on the hub (cluster-scoped)
+   - Creates a `ClusterResourceOverride` with per-cluster customization rules
+     - Each cluster gets a unique `reportNamespace`: `fleet-member-<cluster-name>`
+   - Creates a `ClusterResourcePlacement` (CRP) with `PickFixed` policy
+     - Targets all clusters in the current stage
+   - KubeFleet propagates the customized `MetricCollector` to each member cluster
+
+3. **Metric Collection on Member Clusters**
+   - Metric-collector controller runs on each member cluster
+   - Every 30 seconds, it:
+     - Queries local Prometheus with PromQL: `workload_health`
+     - Prometheus returns metrics for all pods with `prometheus.io/scrape: "true"` annotation
+     - Extracts workload health (1.0 = healthy, 0.0 = unhealthy)
+     - Creates/updates `MetricCollectorReport` on hub in `fleet-member-<cluster-name>` namespace
+   
+4. **Health Evaluation**
+   - Approval-request-controller monitors `MetricCollectorReports` from all stage clusters
+   - Every 15 seconds, it:
+     - Fetches the `WorkloadTracker` to know which workloads to check
+     - For each cluster in the stage:
+       - Reads its `MetricCollectorReport` from `fleet-member-<cluster-name>` namespace
+       - Verifies all tracked workloads are present and healthy
+     - If any workload is missing or unhealthy, waits for next cycle
+     - If ALL workloads across ALL clusters are healthy:
+       - Sets ApprovalRequest condition `Approved: True`
+       - KubeFleet proceeds to roll out the stage
+
+5. **Stage Progression**
+   - KubeFleet applies the update to the approved stage clusters
+   - Creates a new ApprovalRequest for the next stage (if any)
+   - The cycle repeats for each stage
+
+### Key Design Decisions
+
+**Why ClusterResourceOverride?**
+- Each member cluster needs to report to a different namespace on the hub
+- The override injects the cluster-specific `reportNamespace` before deployment
+- This allows a single MetricCollector definition to work across all clusters
+
+**Why PickFixed Placement Policy?**
+- Stages may target different subsets of clusters
+- PickFixed ensures MetricCollector only deploys to clusters in the current stage
+- Avoids collecting metrics from clusters not involved in the stage
+
+**Why 15-second polling for approval?**
+- Balances responsiveness with control plane load
+- Gives clusters time to stabilize after rollout
+- Allows detection of workload health degradation
+
+**Why cluster-scoped MetricCollector?**
+- Simplifies propagation via CRP (no namespace matching issues)
+- Single resource definition covers all namespaces
+- Consistent with KubeFleet's placement model
+
 ## Prerequisites
 
 - Docker or Podman for building images
@@ -264,22 +351,19 @@ kubectl get clusterapprovalrequest -A
 kubectl describe clusterapprovalrequest <approval-request-name>
 ```
 
-## How It Works
-
-1. **Metric Collection**: The standalone-metric-collector on each member cluster queries Prometheus for `workload_health` metrics
-2. **Report Creation**: Collectors create MetricCollectorReport resources on the hub cluster
-3. **Health Monitoring**: The approval-request-controller watches ApprovalRequest resources and corresponding MetricCollectorReports
-4. **Automatic Approval**: When all workloads meet health thresholds defined in WorkloadTracker specs, the controller approves the staged update
-
 ## Configuration
 
 ### Approval Request Controller
 - Located in `approval-request-controller/charts/approval-request-controller/values.yaml`
 - Key settings: log level, resource limits, RBAC, CRD installation
+- Default Prometheus URL: `http://prometheus.prometheus.svc.cluster.local:9090`
+- Reconciliation interval: 15 seconds
 
 ### Metric Collector
-- Located in `standalone-metric-collector/charts/metric-collector/values.yaml`
-- Key settings: hub cluster URL, Prometheus URL, member cluster name, sync interval
+- Located in `metric-collector/charts/metric-collector/values.yaml`
+- Key settings: hub cluster URL, Prometheus URL, member cluster name
+- Metric collection interval: 30 seconds
+- Connects to hub using service account token
 
 ## Troubleshooting