Receiving "Failed to set admission audit annotation... annotation key has invalid format, the right format is a DNS subdomain prefix and '/' and key name."

[erplsf]

Greetings,

I've deployed this project to AWS EKS v1.21, and noticed that the webhook pod is issuing valid AdmissionReviews, it seems that have field allowed: true, but the API server doesn't accept them. So I've looked in Cloudwatch Logs and found the following error messages:

W1207 11:32:02.918031      11 dispatcher.go:268] Failed to set admission audit annotation namespace-node-affinity.namespace-node-affinity.svc/namespace-node-affinity.idgenchev.github.com/applied-patch to [
{
    "op": "add",
    "path": "/spec/affinity",
    "value": {
        "nodeAffinity": {
            "requiredDuringSchedulingIgnoredDuringExecution": {
                "nodeSelectorTerms": [
                    {
                        "matchExpressions": [
                            {
                                "key": "dedicated",
                                "operator": "Equal"
                            }
                        ]
                    }
                ]
            }
        }
    }
}
,
{
    "op": "add",
    "path": "/spec/tolerations/-",
    "value": {
        "key": "dedicated",
        "operator": "Equal",
        "value": "critical",
        "effect": "NoSchedule"
    }
}
] for mutating webhook namespace-node-affinity.namespace-node-affinity.svc: annotation key has invalid format, the right format is a DNS subdomain prefix and '/' and key name. (e.g. 'podsecuritypolicy.admission.k8s.io/admit-policy')

EDIT: and the new pods do not appear in cluster, so with this hook deployed, if it tries to apply configuration, new pod definition will be rejected and not created in cluster, so this is a blocking bug, which stops us from using this useful piece of software.

[erplsf]

Seems to be a PEBKAC, sorry. Turns out you can't use operator: Equal in matchExpressions for nodeSelectorTerms - after i switched to using operator: In, and values: [...] it worked.

[erplsf]

Okay, I'm sorry - it turns out the problem still exists. It's just that it doesn't happen on every pod/namespace:

W1207 11:54:47.015951      11 dispatcher.go:268] Failed to set admission audit annotation namespace-node-affinity.namespace-node-affinity.svc/namespace-node-affinity.idgenchev.github.com/applied-patch to [
{
    "op": "add",
    "path": "/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/-",
    "value": [
        {
            "matchExpressions": [
                {
                    "key": "dedicated",
                    "operator": "In",
                    "values": [
                        "critical"
                    ]
                }
            ]
        }
    ]
}
,
{
    "op": "add",
    "path": "/spec/tolerations/-",
    "value": {
        "key": "dedicated",
        "operator": "Equal",
        "value": "critical",
        "effect": "NoSchedule"
    }
}
] for mutating webhook namespace-node-affinity.namespace-node-affinity.svc: annotation key has invalid format, the right format is a DNS subdomain prefix and '/' and key name. (e.g. 'podsecuritypolicy.admission.k8s.io/admit-policy')

[idgenchev]

Hi @erplsf thanks for raising this!

Looks like the annotation key that the webhook is trying to set here has the wrong format. I'll look into that, meanwhile it'll be super useful if you can share a bit more info around your setup so I can try and reproduce this. For example is the webhook deployed on the same cluster, is it in the same namespace as the one it's enabled for, etc.

[erplsf]

Sure, absolutely!
webhook is deployed on the same cluster, in it's own namespace, configmap for it also lives in the same namespace. I use the default deployment with init container that creates CA bundle for me, and I pin the images to latest release - v2.1.1. I have configured two separate, different namespaces with same config - the idea that I have is to reserve my nodes for all pods in some namespace. In one namespace there's Istio system pods, in the other openfaas related ones.

[erplsf]

and what's more interesting, pods in one namespace (openfaas) are allowed and created, and in (istio) they don't - I think this is related to the fact that the original pods also have preconfigured nodeAffinity and webhook somehow conflicts with it.

[idgenchev]

Thanks for the info @erplsf! The namespace-node-affinity webhook should be able to merge the existing nodeAffinity with what you have configured in the ConfigMap. I'll try to reproduce it locally now.

[idgenchev]

Hey @erplsf looks like you have hit a bug with the tool that @ca-scribner is working on to fix as part of #31. I'll be pushing a new release as soon as that's merged.

Having said that, from the error message it looks like the annotationKey has an incorrect format, so I'll spend some time to see if I can reproduce this anyway :)