Perpetual drift when including OIDC root CA thumbprint is disabled

[mindw]

Description

If the cluster is created with include_oidc_root_ca_thumbprint set to false, The next plan will suggest removing the internally fetched thumbprint.

• ✋ I have searched the open/closed issues and my issue is not listed.

Versions

• Module version [Required]:

$ terraform modules

Modules declared by configuration:
.
├── "eks_bottlerocket"[registry.terraform.io/terraform-aws-modules/eks/aws] 21.10.0 (~> 21.0)
│   ├── "self_managed_node_group"[./modules/self-managed-node-group]
│   │   └── "user_data"[../_user_data]
│   ├── "eks_managed_node_group"[./modules/eks-managed-node-group]
│   │   └── "user_data"[../_user_data]
│   ├── "fargate_profile"[./modules/fargate-profile]
│   └── "kms"[registry.terraform.io/terraform-aws-modules/kms/aws] 4.0.0
└── "vpc"[registry.terraform.io/terraform-aws-modules/vpc/aws] 6.5.1 (~> 6.0)

• Terraform version:

Terraform v1.14.0
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v6.23.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.7
+ provider registry.terraform.io/hashicorp/null v3.2.4
+ provider registry.terraform.io/hashicorp/time v0.13.1
+ provider registry.terraform.io/hashicorp/tls v4.1.0

• Provider version(s):

$ terraform providers -version
Terraform v1.14.0
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v6.23.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.7
+ provider registry.terraform.io/hashicorp/null v3.2.4
+ provider registry.terraform.io/hashicorp/time v0.13.1
+ provider registry.terraform.io/hashicorp/tls v4.1.0

Reproduction Code [Required]

Used examples/self-managed-node-group as is except for
examples/self-managed-node-group/eks-bottlerocket.tf which was modified by adding include_oidc_root_ca_thumbprint = false at line 23:

module "eks_bottlerocket" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  name               = "${local.name}-bottlerocket"
  kubernetes_version = "1.33"

  # EKS Addons
  addons = {
    coredns = {}
    eks-pod-identity-agent = {
      before_compute = true
    }
    kube-proxy = {}
    vpc-cni = {
      before_compute = true
    }
  }

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  include_oidc_root_ca_thumbprint = false
  self_managed_node_groups = {
    example = {
      ami_type      = "BOTTLEROCKET_x86_64"
      instance_type = "m6i.large"

      min_size = 2
      max_size = 5
      # This value is ignored after the initial creation
      # https://github.com/bryantbiggs/eks-desired-size-hack
      desired_size = 2

      # This is not required - demonstrates how to pass additional configuration
      # Ref https://bottlerocket.dev/en/os/1.19.x/api/settings/
      bootstrap_extra_args = <<-EOT
        # The admin host container provides SSH access and runs with "superpowers".
        # It is disabled by default, but can be disabled explicitly.
        [settings.host-containers.admin]
        enabled = false

        # The control host container provides out-of-band access via SSM.
        # It is enabled by default, and can be disabled if you do not expect to use SSM.
        # This could leave you with no way to access the API and change settings on an existing node!
        [settings.host-containers.control]
        enabled = true

        # extra args added
        [settings.kernel]
        lockdown = "integrity"
      EOT
    }
  }

  tags = local.tags
}

Steps to reproduce the behavior:

1. drop the above code as is on top of examples/self-managed-node-group/eks-bottlerocket.tf
2. cd examples/self-managed-node-group/
3. terraform apply --auto-approve
4. terraform plan

Expected behavior

Plan should be empty - like so:

No changes. Your infrastructure matches the configuration.

Actual behavior

Plan shown one change - removing the auto provisioned thumbprint like so:

...
module.eks_bottlerocket.aws_eks_addon.this["kube-proxy"]: Refreshing state... [id=ex-self-mng-bottlerocket:kube-proxy]
module.eks_bottlerocket.aws_eks_addon.this["coredns"]: Refreshing state... [id=ex-self-mng-bottlerocket:coredns]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.eks_bottlerocket.aws_iam_openid_connect_provider.oidc_provider[0] will be updated in-place
  ~ resource "aws_iam_openid_connect_provider" "oidc_provider" {
        id              = "arn:aws:iam::124117252276:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/F8E0A66E6E7363D486CAD003AF7CE302"
        tags            = {
            "Example"    = "ex-self-mng"
            "GithubOrg"  = "terraform-aws-modules"
            "GithubRepo" = "terraform-aws-eks"
            "Name"       = "ex-self-mng-bottlerocket-eks-irsa"
        }
      ~ thumbprint_list = [
          - "06b25927c42a721631c1efd9431e648fa62e1e39",
        ]
        # (4 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
...

Terminal Output Screenshot(s)

see above

Additional context

As per documentation, the thumbprint_list must not exist - an empty list is a list. https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_openid_connect_provider#thumbprint_list-1

One can argue that this needs to be the default but that would be a breaking change.

Suggested fix is in #3586

[bryantbiggs]

its not clear what you are trying to do - I believe the diff is coming from EKS adding the thumbprint automatically (which we matched with the data source here to ensure the diff was clean)

[mindw]

@bryantbiggs thank you for looking into this!

I am trying to use the IAM openid connect provider "auto" mode where thumbprints are managed automatically. See https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_openid_connect_provider#thumbprint_list-1 .
As mentioned in the PR, when null is used (instead of an empty list) there is no drift.

Not sure which data source is being referred - tls_certificate.this count is 0 when var.include_oidc_root_ca_thumbprint is false.

[bryantbiggs]

I am trying to use the IAM openid connect provider "auto" mode where thumbprints are managed automatically.

Why - what is your root intent or motivation? What is leading you down this path?

Also, in a separate thought - why use IRSA instead of the new and improved EKS Pod Identity?

[mindw]

Why - what is your root intent or motivation? What is leading you down this path?

For one each time the thumbprint is updated is creates a cascade of resource updates. Yes, it was every few months but with 20-30 clusters this is quite the pain.

Also, in a separate thought - why use IRSA instead of the new and improved EKS Pod Identity?

We're using it. There is one use case which EKS Pod identity doesn't cover though. This is when a role should apply multiple service accounts based on a wildcard - we use that to cover Kubernetes based application tests.

AFAIK some AWS addons required IMDS IRSA enabled (at least it used too)

Hopefully the above makes sense.

[bryantbiggs]

FYI - This has nothing to do with IMDS

The reason for this is still unclear and I'm leaning towards closing unless there is a clear and valid reason, with evidence, that says otherwise

[mindw]

FYI - This has nothing to do with IMDS

My apologies, I meant IRSA.

I'm not entirely sure why is such a resistance for a backwards compatible bug fix 😕 . After all, include_oidc_root_ca_thumbprint = false is part of the module API.

[bryantbiggs]

I'm not entirely sure why is such a resistance for a backwards compatible bug fix 😕 . After all, include_oidc_root_ca_thumbprint = false is part of the module API.

1. Its not clear if there is a bug. If there was, I would expect more folks chiming in given how much this module is used
2. A change that you think is correct for your case may not be broadly applicable, or even correct at all. I don't know what the ramifications are of this change or if its even a correct solution. As a maintainer, if this causes knock on effects or leads people to footguns, I have to answer for it - if I don't understand it, I can't really answer it other than "one user said they needed this 🤷🏽 "

and in the same vein, I'm not entirely sure why there is such resistance to explaining the goals and motivations (not just the change itself). Why would someone want to enable IRSA, create an IAM OpenID connect provider, and then remove the clusters root CA thumbprint.

[mindw]

@bryantbiggs thank you for your patience and time invested in this. much appreciated!

1. Its not clear if there is a bug. If there was, I would expect more folks chiming in given how much this module is used

The bug is the perpetual drift/diff. Applying the suggested change has no effect. The change will appear in every subsequent plan.
As it seems the IAM provider restores it (I've tested by creating a pod using a role via IRSA and running s3 commands before and after applying).
The module has many, many features, I doubt anyone is aware of this feature except of the original author and the maintainers. I only stumbled on it when investigating un-expected changes appearing even when terraform files were unchanged.

Why would someone want to enable IRSA, create an IAM OpenID connect provider, and then remove the clusters root CA thumbprint.

That's not the use case. Although I did test it - without the patch there is also perpetual drift. The thumbprint list is reverted by the provider to its previous state (more on this below).

The use case is to enable IRSA (which is the default) and create an IAM OpenID provider in "self managed mode" by NOT passing it any thumbprints (done by This is done by setting include_oidc_root_ca_thumbprint = false).
Is this mode the data source is redundant and the provider resource will never be updated.

2. A change that you think is correct for your case may not be broadly applicable, or even correct at all. I don't know what the ramifications are of this change or if its even a correct solution. As a maintainer, if this causes knock on effects or leads people to footguns, I have to answer for it - if I don't understand it, I can't really answer it other than "one user said they needed this 🤷🏽 "

My apologies, It had no intent to suggest otherwise.

The following there cases were tested with vanilla module:

1. include_oidc_root_ca_thumbprint = true (default) - no drift.
2. include_oidc_root_ca_thumbprint = false on creation - there is perpetual drift in module.eks_bottlerocket.aws_iam_openid_connect_provider.oidc_provider[0].
3. include_oidc_root_ca_thumbprint = false after creation - there is also perpetual drift module.eks_bottlerocket.aws_iam_openid_connect_provider.oidc_provider[0].

With the patch:

1. no change
2. no drift
3. no drift - but the list isn't cleared.

From the above and the documentation is seems the thumbprint list is completely ignored when for a provider using the EKS endpoint.

For certain OIDC identity providers (e.g., Auth0, GitHub, GitLab, Google, or those using an Amazon S3-hosted JWKS endpoint), AWS relies on its own library of trusted root certificate authorities (CAs) for validation instead of using any configured thumbprints. In these cases, any configured thumbprint_list is retained in the configuration but not used for verification.

[mindw]

It seems I've accidently closed the issue - and I can't re-open it :(