-
Notifications
You must be signed in to change notification settings - Fork 810
Add support for opensearch searchable snapshot metrics #860
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for opensearch searchable snapshot metrics #860
Conversation
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
…trics Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
e80b455 to
10c519a
Compare
The error itself already has the sanitized URL so there is no need to log the sensitive input string. Signed-off-by: Joe Adams <github@joeadams.io>
|
@sysadmind is there anything missing in this PR to review and merge it? |
Signed-off-by: Evan Tepsic <84512952+etgraylog@users.noreply.github.com>
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5.4.0 to 6.0.0. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@0aaccfd...4469467) --- updated-dependencies: - dependency-name: actions/setup-go dependency-version: 6.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the aws group with 4 updates: [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2), [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2), [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) and [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2). Updates `github.com/aws/aws-sdk-go-v2` from 1.37.2 to 1.39.2 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@v1.37.2...v1.39.2) Updates `github.com/aws/aws-sdk-go-v2/config` from 1.30.3 to 1.31.12 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@v1.30.3...config/v1.31.12) Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.18.3 to 1.18.16 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@config/v1.18.3...config/v1.18.16) Updates `github.com/aws/aws-sdk-go-v2/service/sts` from 1.36.0 to 1.38.6 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@v1.36.0...service/sts/v1.38.6) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2 dependency-version: 1.39.2 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: aws - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-version: 1.31.12 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: aws - dependency-name: github.com/aws/aws-sdk-go-v2/credentials dependency-version: 1.18.16 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws - dependency-name: github.com/aws/aws-sdk-go-v2/service/sts dependency-version: 1.38.6 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: aws ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/checkout](https://github.com/actions/checkout) from 4.2.2 to 5.0.0. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@11bd719...08c6903) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 5.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…theus-community#1084) Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.23.0 to 1.23.2. - [Release notes](https://github.com/prometheus/client_golang/releases) - [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md) - [Commits](prometheus/client_golang@v1.23.0...v1.23.2) --- updated-dependencies: - dependency-name: github.com/prometheus/client_golang dependency-version: 1.23.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [github.com/prometheus/exporter-toolkit](https://github.com/prometheus/exporter-toolkit) from 0.14.0 to 0.14.1. - [Release notes](https://github.com/prometheus/exporter-toolkit/releases) - [Commits](prometheus/exporter-toolkit@v0.14.0...v0.14.1) --- updated-dependencies: - dependency-name: github.com/prometheus/exporter-toolkit dependency-version: 0.14.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
…tized-url Do not log input URL string
…y/dependabot/go_modules/github.com/prometheus/exporter-toolkit-0.14.1 Bump github.com/prometheus/exporter-toolkit from 0.14.0 to 0.14.1
Bumps [github.com/prometheus/common](https://github.com/prometheus/common) from 0.65.0 to 0.66.1. - [Release notes](https://github.com/prometheus/common/releases) - [Changelog](https://github.com/prometheus/common/blob/main/CHANGELOG.md) - [Commits](prometheus/common@v0.65.0...v0.66.1) --- updated-dependencies: - dependency-name: github.com/prometheus/common dependency-version: 0.66.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
…y/dependabot/go_modules/github.com/prometheus/common-0.66.1 Bump github.com/prometheus/common from 0.65.0 to 0.66.1
* Update minimum Go version to 1.24.0. * Update build to Go 1.25. * Cleanup disabled CircleCI pipeline. Signed-off-by: SuperQ <superq@gmail.com>
Signed-off-by: prombot <prometheus-team@googlegroups.com>
* Fix mixed value types for disk watermark The disk watermark values can be a ratio or percentage according to the docs[1], however when set to a percentage, the defaults become an object and therefore fails to parse. In that case, we really only care about what the user set. Adds a test to confirm a fix for prometheus-community#1044. Fixes prometheus-community#1044 [1] https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings#disk-based-shard-allocation Signed-off-by: Joe Adams <github@joeadams.io> * Add missing test fixture Signed-off-by: Joe Adams <github@joeadams.io> --------- Signed-off-by: Joe Adams <github@joeadams.io>
…y/repo_sync Synchronize common files from prometheus/prometheus
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
…ot-filecache' into add-opensearch-searchable-snapshot-filecache # Conflicts: # collector/nodes.go # collector/nodes_response.go
…ch_filecache_active_percent
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
…ch_filecache_active_percent
…ot-filecache' into add-opensearch-searchable-snapshot-filecache
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
…ch_filecache_active_percent Signed-off-by: etgraylog <evan.tepsic@graylog.com>
…ot-filecache' into add-opensearch-searchable-snapshot-filecache
|
Perhaps this enhancement can be attempted again in a new Pull-request, but the commit history at this point from unsigned commits causing failed attempts to rebase I think is unacceptable. |
|
I think that the branch needs to be rebased. It shouldn't be 72 commits. I would think it should be more like 1 or 2 commits depending on the complexity. There's a lot in here that isn't actually this change. A git rebase should fix that. |
Description
Elasticsearch & Opensearch both have a Searchable Snapshots feature that emit metrics from unique API endpoints:
The Elasticsearch Searchable Snapshots metrics appear to be available via the
es.snapshotsargument, while there does not yet appear to be a similar argument for Opensearch Searchable Snapshot metrics.A minimal amount of changes were realized to add support for these metrics, leveraging existing structs & other data types with a new node role while adding necessary elements to for example
nodeMetrics(collector/nodes.go).Motivation and Context
The motivation of this PR is to add support for Opensearch Searchable Snapshots to the
elasticsearch_exporterso that they can be scraped by Prometheus for monitoring & serviceability.Several modifications have been made in a fork to quickly add support for these. A better way to emit these more than likely exists so that for example only nodes with
es_search_node=truewould be collected. Moreover, adding a CLI argument to enable these metrics as they are not applicable to Elasticsearch. The changes in this PR could be a decent starting place however if not at least found useful by others.Details
The proposed changes add 8x new metrics to the collection from the Opensearch Nodes API: