Elasticsearch credentials rotation on k8s

[bozho]

Hi,

We are deploying the exporter using the community helm chart and are passing in ES credentials as env. variables (ES_USERNAME and ES_PASSWORD), with values picked up from a Secret.

This makes credentials rotation quite cumbersome, since elasticsearch-exporter deployment needs to be restarted when credentials have been changed and the referenced Secret changes.

Would it be possible to add support for loading credentials from a file (like ES does with file authentication realm), with automatic reload on change? This would allow mounting the credentials Secret as a file, and updating the secret would update the mounted file. This would vastly simplify automating credentials rotation (using external-secrets controller, for example).

[AstralJaeger]

Why not use something like stakater/Reloader?

[bozho]

@AstralJaeger mainly because of a possible memory leak issue with it.

I've solved the issue for now using a kyverno policy (we already use it), which triggers on a Secret change and sets an annotation with the secret's hash on elasticsearch-exporter Deployment's pod template. Changing pod template's annotations triggers a rolling update for the deployment.

The exporter will be unable to connect for a short interval after external-secrets changes the password, ES picks it up and exporter's pods get restarted, but 10 seconds every 90 days is acceptable at this point.

[AstralJaeger]

I see, thats odd, we don't have any such issues and it works wonderfully, but if you were able to reuse an existing tool than thats great news!