"error": "resource manager factory not found"

[rtiwari-ui]

Hi There, I am trying to adopt the scaling policy & scalable target from one eks cluster to another, after adopting it on another cluster, I am getting this error in ack-controller logs:

2025-03-10T13:00:05.584Z ERROR Reconciler error {"controller": "adoptedresource", "controllerGroup": "services.k8s.aws", "controllerKind": "AdoptedResource", "AdoptedResource": {"name":"ack-scalable-target-custom-test-policy","namespace":"test-scaling"}, "namespace": "test-scaling", "name": "ack-scalable-target-custom-test-policy", "reconcileID": "122a6e29-32a0-346a-bb82-74bd0d1ffc67", "error": "resource manager factory not found"}

The describe output of scaling-policy & scalable-target both are below:

Conditions: Last Transition Time: 2025-03-10T13:58:49Z Message: Resource already exists Reason: This resource already exists but is not managed by ACK. To bring the resource under ACK management, you should explicitly adopt the resource by creating a services.k8s.aws/AdoptedResource Status: True Type: ACK.Terminal Last Transition Time: 2025-03-10T13:58:49Z Message: Resource not synced Reason: resource is in terminal condition Status: False Type: ACK.ResourceSynced

These are the ack-controller & application-autoscaling controller version:

Application-autoscaling: public.ecr.aws/aws-controllers-k8s/applicationautoscaling-controller:v1.0.2
ACK-controller: public.ecr.aws/aws-controllers-k8s/sagemaker-controller:v1.2.0

[github-actions]

Hello @RajDevO 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[michaelhtm]

Hi @RajDevO, can you post the adoption manifest you're using?

[rtiwari-ui]

Hi @michaelhtm here is the manifest that I am using:

Adopt scalable targets

'apiVersion: services.k8s.aws/v1alpha1
kind: AdoptedResource
metadata:
  name: ack-scalable-target-custom-test-target
  annotations:
    services.k8s.aws/region: us-west-2
spec:
  aws:
    nameOrID: ack-scalable-target-custom-test-target
  kubernetes:
    group: sagemaker.services.k8s.aws
    kind: ScalableTarget
    metadata:
      name: ack-scalable-target-custom-test-target'

Adopt scaling policy

'apiVersion: services.k8s.aws/v1alpha1
kind: AdoptedResource
metadata:
  name: ack-scalable-target-custom-test-policy
  annotations:
    services.k8s.aws/region: us-west-2
spec:
  aws:
    nameOrID: ack-scalable-target-custom-test-policy
  kubernetes:
    group: sagemaker.services.k8s.aws
    kind: ScalingPolicy
    metadata:
      name: ack-scalable-target-custom-test-policy`

[michaelhtm]

Hi @RajDevO, thank you for providing the manifest. We no longer recommend using the AdoptedResource CRD to adopt resources. We introduced a new feature that allows resource adoption by annotation. More info here: https://aws-controllers-k8s.github.io/community/docs/user-docs/features/#resourceadoption

I also noticed you're using sagemaker.services.k8s.aws instead of applicationautoscaling.services.k8s.aws

Here's what it would look like

apiVersion: applicationautoscaling.services.k8s.aws/v1alpha1
kind: ScalableTarget
metadata:
  name: ack-scalable-target-custom-test-target
  annotations:
    services.k8s.aws/adoption-policy: adopt
    services.k8s.aws/adoption-fields: |
      {
          "resourceID": "<your-resource-ID"
          # This is a required field
          "serviceNamespace": "<your-serviceNamespace"
          "scalableDimension": "<your-scalableDimension"
      }

apiVersion: applicationautoscaling.services.k8s.aws/v1alpha1
kind: ScalingPolicy
metadata:
  name: ack-scalable-target-custom-test-policy
  annotations:
    services.k8s.aws/adoption-policy: adopt
    services.k8s.aws/adoption-fields: |
      {
          "resourceID": "<your-resource-ID"
          # This is a required field
          "serviceNamespace": "<your-serviceNamespace"
          "scalableDimension": "<your-scalableDimension"
      }

[rtiwari-ui]

Hi @michaelhtm, thanks it worked for scalable targets, also have just a question is this aloha features are applicable for endpoint, endpointconfigs & models?

Also even after fixing the issue the error logs are still showing below message, I also have recycled the old pods of ack-controller & autoscaling controller.
2025-03-10T13:00:05.584Z ERROR Reconciler error {"controller": "adoptedresource", "controllerGroup": "services.k8s.aws", "controllerKind": "AdoptedResource", "AdoptedResource": {"name":"ack-scalable-target-custom-test-policy","namespace":"test-scaling"}, "namespace": "test-scaling", "name": "ack-scalable-target-custom-test-policy", "reconcileID": "122a6e29-32a0-346a-bb82-74bd0d1ffc67", "error": "resource manager factory not found"}

[michaelhtm]

@intersteller-loom Hello,
The new feature is applicable to all ACK controllers!

from the logs I can see the AdoptedResource CR is still trying to be reconciled. The message resource manager not found implies that the resource kind you are trying to create is not supported in the controller.

For example using the sagemake.services.k8s.awsr group, which triggers the sagemaker-controller reconciliation loop, which does not recognize the ScalingPolicy and ScalingTarget types, which are resources supported in the applicationautoscaling controller, so the group would need to be applicationautoscaling.services.k8s.aws

[rtiwari-ui]

Hi @michaelhtm I was looking for any example for adopting the resources such as Endpoint, EndpointConfig, Model because I couldn't find any, my confusion is what we will be using for "Adoption-field" under annotation, it should ARN or Name of resource ?because we don't have anything like "Resource id" for endpoint, endpointconfig & model.

Could you please share an example like below?

apiVersion: applicationautoscaling.services.k8s.aws/v1alpha1
kind: ScalingPolicy
metadata:
  name: ack-scalable-target-custom-test-policy
  annotations:
    services.k8s.aws/adoption-policy: adopt
    services.k8s.aws/adoption-fields: |
      {
          "resourceID": "<your-resource-ID"
          # This is a required field
          "serviceNamespace": "<your-serviceNamespace"
          "scalableDimension": "<your-scalableDimension"
      }

[michaelhtm]

HI @intersteller-loom
We currently don't have documentation for the required adoption fields..we have a github issue mentioning the need for an automated way of publishing such fields. For now the best way to track these fields is here:
https://github.com/aws-controllers-k8s/sagemaker-controller/blob/8b7855dfe7a9d3a85d2dc9535c82f8b60afde4c5/pkg/resource/endpoint/resource.go#L97-L103
https://github.com/aws-controllers-k8s/sagemaker-controller/blob/8b7855dfe7a9d3a85d2dc9535c82f8b60afde4c5/pkg/resource/model/resource.go#L97-L105

In every controller, in pkg/resource/<RESOURCE_NAME>/resource.go there should be a method PopulateResourceFromAnnotation where you can find the fields the controller is looking for :)

Endpoint Example:

apiVersion: sagemaker.services.k8s.aws/v1alpha1
kind: Endpoint
metadata:
  name: $ENDPOINT_NAME
  annotations:
    services.k8s.aws/adoption-policy: adopt
    services.k8s.aws/adoption-fields: |
      {
          "endpointName": "<YOUR_ENDPOINT_NAME>"
      }

[rtiwari-ui]

Thanks @michaelhtm I have tested the above suggestions and getting positive results.

[michaelhtm]

That's great @rtiwari-ui
Feel free to close this issue if it's resolved. Thanks!