Perpetual drift when including OIDC root CA thumbprint is disabled (v2)

[mindw]

Description

Accidently closed #3601

If the cluster is created with include_oidc_root_ca_thumbprint set to false, The next plan will suggest removing the internally fetched thumbprint.

• ✋ I have searched the open/closed issues and my issue is not listed.

Versions

• Module version [Required]:

$ terraform modules

Modules declared by configuration:
.
├── "eks_bottlerocket"[registry.terraform.io/terraform-aws-modules/eks/aws] 21.10.0 (~> 21.0)
│   ├── "self_managed_node_group"[./modules/self-managed-node-group]
│   │   └── "user_data"[../_user_data]
│   ├── "eks_managed_node_group"[./modules/eks-managed-node-group]
│   │   └── "user_data"[../_user_data]
│   ├── "fargate_profile"[./modules/fargate-profile]
│   └── "kms"[registry.terraform.io/terraform-aws-modules/kms/aws] 4.0.0
└── "vpc"[registry.terraform.io/terraform-aws-modules/vpc/aws] 6.5.1 (~> 6.0)

• Terraform version:

Terraform v1.14.0
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v6.23.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.7
+ provider registry.terraform.io/hashicorp/null v3.2.4
+ provider registry.terraform.io/hashicorp/time v0.13.1
+ provider registry.terraform.io/hashicorp/tls v4.1.0

• Provider version(s):

$ terraform providers -version
Terraform v1.14.0
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v6.23.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.7
+ provider registry.terraform.io/hashicorp/null v3.2.4
+ provider registry.terraform.io/hashicorp/time v0.13.1
+ provider registry.terraform.io/hashicorp/tls v4.1.0

Reproduction Code [Required]

Used examples/self-managed-node-group as is except for
examples/self-managed-node-group/eks-bottlerocket.tf which was modified by adding include_oidc_root_ca_thumbprint = false at line 23:

module "eks_bottlerocket" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  name               = "${local.name}-bottlerocket"
  kubernetes_version = "1.33"

  # EKS Addons
  addons = {
    coredns = {}
    eks-pod-identity-agent = {
      before_compute = true
    }
    kube-proxy = {}
    vpc-cni = {
      before_compute = true
    }
  }

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  include_oidc_root_ca_thumbprint = false
  self_managed_node_groups = {
    example = {
      ami_type      = "BOTTLEROCKET_x86_64"
      instance_type = "m6i.large"

      min_size = 2
      max_size = 5
      # This value is ignored after the initial creation
      # https://github.com/bryantbiggs/eks-desired-size-hack
      desired_size = 2

      # This is not required - demonstrates how to pass additional configuration
      # Ref https://bottlerocket.dev/en/os/1.19.x/api/settings/
      bootstrap_extra_args = <<-EOT
        # The admin host container provides SSH access and runs with "superpowers".
        # It is disabled by default, but can be disabled explicitly.
        [settings.host-containers.admin]
        enabled = false

        # The control host container provides out-of-band access via SSM.
        # It is enabled by default, and can be disabled if you do not expect to use SSM.
        # This could leave you with no way to access the API and change settings on an existing node!
        [settings.host-containers.control]
        enabled = true

        # extra args added
        [settings.kernel]
        lockdown = "integrity"
      EOT
    }
  }

  tags = local.tags
}

Steps to reproduce the behavior:

1. drop the above code as is on top of examples/self-managed-node-group/eks-bottlerocket.tf
2. cd examples/self-managed-node-group/
3. terraform apply --auto-approve
4. terraform plan

Expected behavior

Plan should be empty - like so:

No changes. Your infrastructure matches the configuration.

Actual behavior

Plan shown one change - removing the auto provisioned thumbprint like so:

...
module.eks_bottlerocket.aws_eks_addon.this["kube-proxy"]: Refreshing state... [id=ex-self-mng-bottlerocket:kube-proxy]
module.eks_bottlerocket.aws_eks_addon.this["coredns"]: Refreshing state... [id=ex-self-mng-bottlerocket:coredns]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.eks_bottlerocket.aws_iam_openid_connect_provider.oidc_provider[0] will be updated in-place
  ~ resource "aws_iam_openid_connect_provider" "oidc_provider" {
        id              = "arn:aws:iam::124117252276:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/F8E0A66E6E7363D486CAD003AF7CE302"
        tags            = {
            "Example"    = "ex-self-mng"
            "GithubOrg"  = "terraform-aws-modules"
            "GithubRepo" = "terraform-aws-eks"
            "Name"       = "ex-self-mng-bottlerocket-eks-irsa"
        }
      ~ thumbprint_list = [
          - "06b25927c42a721631c1efd9431e648fa62e1e39",
        ]
        # (4 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
...

Terminal Output Screenshot(s)

see above

Additional context

As per documentation, the thumbprint_list must not exist - an empty list is a list. https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_openid_connect_provider#thumbprint_list-1

One can argue that this needs to be the default but that would be a breaking change.

Suggested fix is in #3586

[mindw]

@bryantbiggs thank you for your patience and time invested in this. much appreciated!

1. Its not clear if there is a bug. If there was, I would expect more folks chiming in given how much this module is used

The bug is the perpetual drift/diff. Applying the suggested change has no effect. The change will appear in every subsequent plan.
As it seems the IAM provider restores it (I've tested by creating a pod using a role via IRSA and running s3 commands before and after applying).
The module has many, many features, I doubt anyone is aware of this feature except of the original author and the maintainers. I only stumbled on it when investigating un-expected changes appearing even when terraform files were unchanged.

Why would someone want to enable IRSA, create an IAM OpenID connect provider, and then remove the clusters root CA thumbprint.

That's not the use case. Although I did test it - without the patch there is also perpetual drift. The thumbprint list is reverted by the provider to its previous state (more on this below).

The use case is to enable IRSA (which is the default) and create an IAM OpenID provider in "self managed mode" by NOT passing it any thumbprints (done by This is done by setting include_oidc_root_ca_thumbprint = false).
Is this mode the data source is redundant and the provider resource will never be updated.

2. A change that you think is correct for your case may not be broadly applicable, or even correct at all. I don't know what the ramifications are of this change or if its even a correct solution. As a maintainer, if this causes knock on effects or leads people to footguns, I have to answer for it - if I don't understand it, I can't really answer it other than "one user said they needed this 🤷🏽 "

My apologies, It had no intent to suggest otherwise.

The following there cases were tested with vanilla module:

1. include_oidc_root_ca_thumbprint = true (default) - no drift.
2. include_oidc_root_ca_thumbprint = false on creation - there is perpetual drift in module.eks_bottlerocket.aws_iam_openid_connect_provider.oidc_provider[0].
3. include_oidc_root_ca_thumbprint = false after creation - there is also perpetual drift module.eks_bottlerocket.aws_iam_openid_connect_provider.oidc_provider[0].

With the patch:

1. no change
2. no drift
3. no drift - but the list isn't cleared.

From the above and the documentation is seems the thumbprint list is completely ignored when for a provider using the EKS endpoint.

For certain OIDC identity providers (e.g., Auth0, GitHub, GitLab, Google, or those using an Amazon S3-hosted JWKS endpoint), AWS relies on its own library of trusted root certificate authorities (CAs) for validation instead of using any configured thumbprints. In these cases, any configured thumbprint_list is retained in the configuration but not used for verification.