Skip to content

Conversation

@etgraylog
Copy link

@etgraylog etgraylog commented Feb 23, 2024

Description

Elasticsearch & Opensearch both have a Searchable Snapshots feature that emit metrics from unique API endpoints:

The Elasticsearch Searchable Snapshots metrics appear to be available via the es.snapshots argument, while there does not yet appear to be a similar argument for Opensearch Searchable Snapshot metrics.

A minimal amount of changes were realized to add support for these metrics, leveraging existing structs & other data types with a new node role while adding necessary elements to for example nodeMetrics (collector/nodes.go).

Motivation and Context

The motivation of this PR is to add support for Opensearch Searchable Snapshots to the elasticsearch_exporter so that they can be scraped by Prometheus for monitoring & serviceability.

Several modifications have been made in a fork to quickly add support for these. A better way to emit these more than likely exists so that for example only nodes with es_search_node=true would be collected. Moreover, adding a CLI argument to enable these metrics as they are not applicable to Elasticsearch. The changes in this PR could be a decent starting place however if not at least found useful by others.

Details

The proposed changes add 8x new metrics to the collection from the Opensearch Nodes API:

Metric name Metric type
elasticsearch_filecache_active_in_bytes Gauge
elasticsearch_filecache_active_percent Gauge
elasticsearch_filecache_evictions_in_bytes Gauge
elasticsearch_filecache_hit_count Gauge
elasticsearch_filecache_miss_count Gauge
elasticsearch_filecache_total_in_bytes Gauge
elasticsearch_filecache_used_in_bytes Gauge
elasticsearch_filecache_used_percent Gauge

Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
…trics

Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
@etgraylog etgraylog force-pushed the add-opensearch-searchable-snapshot-filecache branch from e80b455 to 10c519a Compare February 23, 2024 23:46
The error itself already has the sanitized URL so there is no need to log the sensitive input string.

Signed-off-by: Joe Adams <github@joeadams.io>
@mnin
Copy link

mnin commented Oct 1, 2025

@sysadmind is there anything missing in this PR to review and merge it?

etgraylog and others added 14 commits October 2, 2025 00:53
Signed-off-by: Evan Tepsic <84512952+etgraylog@users.noreply.github.com>
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5.4.0 to 6.0.0.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](actions/setup-go@0aaccfd...4469467)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-version: 6.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the aws group with 4 updates: [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2), [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2), [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) and [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2).


Updates `github.com/aws/aws-sdk-go-v2` from 1.37.2 to 1.39.2
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](aws/aws-sdk-go-v2@v1.37.2...v1.39.2)

Updates `github.com/aws/aws-sdk-go-v2/config` from 1.30.3 to 1.31.12
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](aws/aws-sdk-go-v2@v1.30.3...config/v1.31.12)

Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.18.3 to 1.18.16
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](aws/aws-sdk-go-v2@config/v1.18.3...config/v1.18.16)

Updates `github.com/aws/aws-sdk-go-v2/service/sts` from 1.36.0 to 1.38.6
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](aws/aws-sdk-go-v2@v1.36.0...service/sts/v1.38.6)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-version: 1.39.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: aws
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-version: 1.31.12
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: aws
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-version: 1.18.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: aws
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-version: 1.38.6
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: aws
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/checkout](https://github.com/actions/checkout) from 4.2.2 to 5.0.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@11bd719...08c6903)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: 5.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…theus-community#1084)

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.23.0 to 1.23.2.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.23.0...v1.23.2)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-version: 1.23.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [github.com/prometheus/exporter-toolkit](https://github.com/prometheus/exporter-toolkit) from 0.14.0 to 0.14.1.
- [Release notes](https://github.com/prometheus/exporter-toolkit/releases)
- [Commits](prometheus/exporter-toolkit@v0.14.0...v0.14.1)

---
updated-dependencies:
- dependency-name: github.com/prometheus/exporter-toolkit
  dependency-version: 0.14.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
…y/dependabot/go_modules/github.com/prometheus/exporter-toolkit-0.14.1

Bump github.com/prometheus/exporter-toolkit from 0.14.0 to 0.14.1
Bumps [github.com/prometheus/common](https://github.com/prometheus/common) from 0.65.0 to 0.66.1.
- [Release notes](https://github.com/prometheus/common/releases)
- [Changelog](https://github.com/prometheus/common/blob/main/CHANGELOG.md)
- [Commits](prometheus/common@v0.65.0...v0.66.1)

---
updated-dependencies:
- dependency-name: github.com/prometheus/common
  dependency-version: 0.66.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
…y/dependabot/go_modules/github.com/prometheus/common-0.66.1

Bump github.com/prometheus/common from 0.65.0 to 0.66.1
* Update minimum Go version to 1.24.0.
* Update build to Go 1.25.
* Cleanup disabled CircleCI pipeline.

Signed-off-by: SuperQ <superq@gmail.com>
Signed-off-by: prombot <prometheus-team@googlegroups.com>
* Fix mixed value types for disk watermark

The disk watermark values can be a ratio or percentage according to the docs[1], however when set to a percentage, the defaults become an object and therefore fails to parse. In that case, we really only care about what the user set. Adds a test to confirm a fix for prometheus-community#1044.

Fixes prometheus-community#1044

[1] https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings#disk-based-shard-allocation

Signed-off-by: Joe Adams <github@joeadams.io>

* Add missing test fixture

Signed-off-by: Joe Adams <github@joeadams.io>

---------

Signed-off-by: Joe Adams <github@joeadams.io>
…y/repo_sync

Synchronize common files from prometheus/prometheus
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
…ot-filecache' into add-opensearch-searchable-snapshot-filecache

# Conflicts:
#	collector/nodes.go
#	collector/nodes_response.go
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
…ot-filecache' into add-opensearch-searchable-snapshot-filecache
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
Signed-off-by: etgraylog <evan.tepsic@graylog.com>
…ch_filecache_active_percent

Signed-off-by: etgraylog <evan.tepsic@graylog.com>
…ot-filecache' into add-opensearch-searchable-snapshot-filecache
@etgraylog
Copy link
Author

Perhaps this enhancement can be attempted again in a new Pull-request, but the commit history at this point from unsigned commits causing failed attempts to rebase I think is unacceptable.

@etgraylog etgraylog closed this Dec 1, 2025
@sysadmind
Copy link
Contributor

I think that the branch needs to be rebased. It shouldn't be 72 commits. I would think it should be more like 1 or 2 commits depending on the complexity. There's a lot in here that isn't actually this change. A git rebase should fix that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants