Deletion of Resources using Deleted Read Only Refs Not Working

[Beardface123]

Describe the bug

Undesirable behavior when working with read-only resources used as a ref to other child resources.

When using the prometheuesservice-controller, I'm able to create a situation where I adopt an AMP workspace as read-only, and then create a rulegroupsnamespace using the workspace as a ref. This works great.

The issue is upon deletion. When cleaning up the resources, if the workspace object is deleted first, the rulegroupsnamespace object gets removed, and finalizers do not clean up the object in AWS.

Steps to reproduce

1. Create AMP workspace out of band, console/cli/api
2. Adopt this workspace as read only with prometheusservice.services.k8s.aws CRD
3. Create a rule groups namespace with prometheusservice.services.k8s.aws CRD
4. Delete the workspace object
5. Delete rulegroupsnamespace object

Observe to see the rulegroupsnamespace object has not been deleted. Confirmed the service controller's retention policy is set to delete.

Expected outcome

I would've expected the rule namespace to be deleted. I directly tested ordering of the deletion. If the rulegroupsnamespace object is deleted first, it is cleaned up correctly.

Environment

• Kubernetes version - 1.28
• Using EKS (yes/no), if so version? Yes, eks.56
• AWS service targeted (S3, RDS, etc.) prometheusservice

** SAMPLE MANIFESTS **

---
apiVersion: prometheusservice.services.k8s.aws/v1alpha1
kind: Workspace
metadata:
  name: amp-workspace-test
  namespace: ack-system
  annotations:
    services.k8s.aws/read-only: "true"
    services.k8s.aws/adoption-policy: "adopt"
    services.k8s.aws/adoption-fields: | 
        {
          "workspaceID": "workspaceIDHere"
        }
---
apiVersion: prometheusservice.services.k8s.aws/v1alpha1
kind: RuleGroupsNamespace
metadata:
  name: amp-workspace-test-rulegroup-test
  namespace: ack-system
spec:
  workspaceRef: 
    from: 
      name: amp-workspace-test
      namespace: ack-system
  configuration: |
    groups:  
    - name: test
      rules:
      - record: somequery:sum
        expr: some expression
  name: test
  tags: {}

[github-actions]

Hello @Beardface123 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[a-hilaly]

@Beardface123 for proper ordering of resources during creation and cleanup with do recommend https://kro.run/ - otherwise ACK doesn't have the concept of ordering (in deletion/creation) and DAGs

[Beardface123]

Hi @a-hilaly ! Apologies if my request/issue wasn't clear. This isn't an issue of "order of operations" in the cluster. This is a problem where lifecycle management of AWS resources isn't consistent.

In the situation outlined above, it seems the controller chose to orphan the Rules Namespace altogether which is completely contrary to the behavior I have followed in this document

https://aws-controllers-k8s.github.io/community/docs/user-docs/deletion-policy/

In my case there is

• no services.k8s.aws/deletion-policy annotation on the resource itself
• no prometheusservice.services.k8s.aws/deletion-policy annotation on the namespace
• the controller is set with --deletion-policy=delete

I would expect a read-only resource to be retained, and the RuleGroupsNamespace to be deleted regardless of order in which the finalizers were triggered.