Guidance

Here is a list of usual customization rules and tips or recommendations possible to use for their detectors configuration. This is not an exhaustive list but should provide key features for most of the users. Detailed explanations how are created and how work this in Templating.

tagging convention

• monitoring config should be as generic as possible and rely on metadata from sources
• every module implement a default tagging convention (in general from user inputs)
• usual logic is filtering in on
  • sfx_monitored:true a common flag to enable alerting on a resource (or ignore some of them)
  • env set from the module common environment variable by user
• this could differ depending on the source of metrics either because dimensions are prefixed (aws_tag...) or not collected / available (newrelic, aws vpn, gcp).
• if constraints do not allow to match this convention feel free to override with custom one

aggregation

• do not aggregate will evaluate every single MTS separately considering every avaiable dimensions values combination. advantage: will apply to every reporting resources no matter the situation and without to know them drawback: will be sensitive to the any dimensions changes (because of a new granularity or the disappearing of a MTS)
• aggregate on a set of dimension(s) allow to "group" multiple MTS into one so this restrict the evaluation to these dimensions only (i.e. mts without one of the dimensions key will be ignored) advantage: the behavior is always the same (grouping and granularity do not change) drawback: we must to know valid and available set of dimensions to define the right group but they highly depend on the env, deployment, config ..
• do not consider dimension(s) is only possible aggregating to other ones (impossible without aggregation).

In general do not aggregate is the most generic and easy way but:

• sometimes we want to evaluate at "higher" level (i.e. not by host but for the entire cluster)
• some use cases could be very sensitive to dimensions / grouping changes (i.e. heartbeat)

heartbeat

• perfect for healthcheck while it will fire alert for every group which do not report anymore.
• but highly depends on aggregation which defines the groups to evaluate and consider as "unhealthy".
• do not aggregate make the implementation generic but will lead to alert for every single disappearing MTS (a simple dimension change will remove old MTS and create a new one).
• on another side, the aggregation group to define is not always the same and could not be an universal default.
• indeed, dimensions could change depending on environment and configuration and even with same dimensions the user could want different alerting granularity (by host, by cluster ..)
• as much as possible modules do not use aggregation which will work for basic scenario
• some modules use aggregation because the monitor provides not relevant too high granularity for heartbeat (i.e. database dimension on postgresql will lead alert for database dropped).
• but in both cases we highly recommend to define the aggregation adapted in your scenario depending of avaialble dimensions and what you expect to montior.
• it is also possible to configure
• vm state are filtered out automatically to support downscaling from gcp, aws and azure.
• when a MTS (when no aggregation) or a group of MTS (when aggregation) disappear and lead to heartbeat alert you need to wait 36h to signalfx consider as inactive and stop to raise alert on it.

notifications

• severities
• levels definition / best practices
• how to mapping with example

agent config tips:

• standard deployment and others (disableHostDimensions + extraDimensions)
• disableEndpointDimensions
• dimensionTransformations
• datapointsToExclude (whitelist filtering)
• service discovery
• kubernetes 2 deployuments
• use globalDimension with caution because it will impact every metrics (and could generate heartbeat alert for each little change)
• use extraDimensions to add scope / context to detectors