Provider does CRD validation at plan time and should do this at apply time.

[jsingleton785]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 1.9.6
Kubernetes provider version: 2.32.0
Kubernetes version: 1.28.13

Affected Resource(s)

kubernetes_manifest

Terraform Configuration Files

resource "helm_release" "cert_manager" {
chart = "/tmp/cert-manager.tgz"
name = "cert-manager"
namespace = "cert-manager"
wait = false
values = {...}
}

resource "kubernetes_manifest" "ClusterIssuer" {
depends_on = [ helm_release.cert_manager ]
manifest = {
apiVersion = "cert-manager.io/v1"
kind = "ClusterIssuer"
metadata = {
name = "issuer"
}
spec = {
selfSigned = {}
}
}
}

### Debug Output
│ Error: API did not recognize GroupVersionKind from manifest (CRD may not be installed)
│
│   with kubernetes_manifest.ClusterIssuer,
│   on main.tf line 64, in resource "kubernetes_manifest" "ClusterIssuer":
│   64: resource "kubernetes_manifest" "ClusterIssuer" {
│
│ no matches for kind "ClusterIssuer" in group "cert-manager.io"

### Steps to Reproduce
1. `terraform apply`

### Expected Behavior
Provider uses the depends_on properly to wait for the other resource to be created, then it will find the CRD properly and the apply will work without error.

If necessary,  it would make sense for this to throw a warning, but not fail the apply, during the plan.

### Actual Behavior
Provider checks for the CRD before the other resource can be created and doesn't allow the module to proceed to the apply step. 

### References
https://github.com/hashicorp/terraform-provider-kubernetes/issues/2187

### Community Note
<!--- Please keep this note for the community --->
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

[arybolovlev]

Hi @jsingleton785,

This is not a bug but how Terraform and kubernetes_manifest work. Terraform initializes all providers at once and one of the first steps that the Kubernetes provider does when the kubernetes_manifest resource is involved is to get CRD for the resource it aims to manage. This happens despite the depends_on option. You either need to separate CRD installation with managing corresponding resources, or use the -target first to install CRD and then manager its resources.

[jsingleton785]

You are correct about this being the way that this provider operates, but this is not how Terraform inherently operates. A provider should not require connection to the running service, in this case a Kubernetes cluster, to run a plan, but the way this is written it does. The plan stage should only compare against the state, and do built in validation of the objects.

The Helm provider doesn’t require access to the cluster to run a plan, and the AWS provider doesn’t require access to AWS to run a plan. So why does this provider? Validation of the actual AWS object by AWS is done at apply time only and that works well.

It may be true that this isn’t a “bug,” but it something that should be addressed. I guess I’ll resubmit it as a feature then. This is one specific use case, but the fact you can’t run a plan without a connection is a larger problem. I should be able to run the plan in a pipeline for MRs to allow proper review of what’s going to change, but it will fail since the provider requires a connection to the cluster even at plan time.

[atlet99]

I’ve decided to bring the discussion back!

The current behavior of the provider, where CRD validation is performed during the plan phase, causes significant issues in workflows. This breaks Terraform’s core logic, where the planning phase should be independent of the state of external systems. This is especially problematic in CI/CD scenarios, where cluster connectivity may not be available during the plan phase.

• the current provider should either skip CRD validation entirely during the plan phase or defer it to the apply phase. This would allow the process to proceed even if the CRD is not yet installed, relying on the dependencies defined in depends_on to ensure proper execution.

This behavior disrupts the expected workflow for infrastructure as code and makes the provider less practical to use. Implementing deferred or skipped CRD validation during the plan phase would resolve these issues and significantly improve usability.

Provide feedback to resolve this issue. Thank you!

[tom10271]

Please just add a skip_validation attribute to let us go. This makes defining OPA policy problematic as policies have to be defined as CRD before they are usable.

[tikolsky]

Is there a way to request this issue to be reopened?

[dm3ch]

This also affects not only CR installation in same module as CRD but also it affects some manifests that may rely on not yet created resources, even if you use depends_on. For example if I'm trying to create Istio destination rule using endpoint got from ElasticCache creation - it's result in try to validate empty string

[nguyenthengocdev]

up

[UladzK]

up! the issue is still reproduced:

╷
│ Error: API did not recognize GroupVersionKind from manifest (CRD may not be installed)
│ 
│   with kubernetes_manifest.eso_infisical_secret_store,
│   on main.tf line 36, in resource "kubernetes_manifest" "eso_infisical_secret_store":
│   36: resource "kubernetes_manifest" "eso_infisical_secret_store" {
│ 
│ no matches for kind "SecretStore" in group "external-secrets.io"
╵

at this moment splitting up the helm releases and dependent manifests into different modules help, but it's clumsy.

[RafaSRibeiro]

│ Error: API did not recognize GroupVersionKind from manifest (CRD may not be installed) │ │ with module.helm_pack.kubernetes_manifest.letsencrypt_cluster_issuer[0], │ on .terraform/modules/helm_pack/helm_cert_manager.tf line 43, in resource "kubernetes_manifest" "letsencrypt_cluster_issuer": │ 43: resource "kubernetes_manifest" "letsencrypt_cluster_issuer" { │ │ no matches for kind "ClusterIssuer" in group "cert-manager.io" ╵