Don't log collector errors #1050
Conversation
collector/collector.go
Outdated
| logger.Debug("collector returned no data", "name", name, "duration_seconds", duration.Seconds(), "err", err) | ||
| } else { | ||
| logger.Error("collector failed", "name", name, "duration_seconds", duration.Seconds(), "err", err) | ||
| logger.Debug("collector failed", "name", name, "duration_seconds", duration.Seconds(), "err", err) |
There was a problem hiding this comment.
I think Error is the right level here. It really is an error.
There was a problem hiding this comment.
No, it's not normal for collectors to error, we have success metrics to let users know that individual collectors fail.
We don't want scrapes to generate logs.
There was a problem hiding this comment.
Users often go to the logs when things aren't working as expected. We have metrics that say was successful or failed, but that doesn't tell you the why. That's what the logs are for.
There was a problem hiding this comment.
It's still pretty standard for exporters to not log individual scrape errors due to the amount of noise that generates.
At a minimum, this should probably be Warn, since it's not fatal to the scrape itself.
There was a problem hiding this comment.
I would be happy to agree on Warn
Avoid logging collector failures at Error level to avoid log spam. Signed-off-by: SuperQ <superq@gmail.com>
SuperQ
force-pushed
the
superq/log_noise
branch
from
ab6089d to
a690241
Compare
Avoid logging collector failures at Error level to avoid log spam. Signed-off-by: SuperQ <superq@gmail.com>
Avoid logging collector failures at Error level to avoid log spam. Signed-off-by: SuperQ <superq@gmail.com> Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Avoid logging collector failures at Error level to avoid log spam.