Skip to content

Conversation

@pincher95
Copy link
Contributor

PR: Add multi-target /probe endpoint + auth-modules (fixes #1062)

This PR implements the feature described in issue #1062 (“multi-target scraping”).

───────────────────────────────
What’s new
───────────────────────────────

  1. /probe endpoint
    Accepts ?target= and (optionally) auth_module=.
    For each request the exporter spins up a throw-away Prometheus registry, registers the usual collectors with the supplied ES URL, gathers, and streams metrics.

  2. Auth-module support (--config.file=)
    Simple YAML schema modelled after postgres_exporter & mysqld_exporter.

    auth_modules:
      prod_basic:
        type: userpass
        userpass:
          username: metrics
          password: s3cr3t
    
      staging_key:
        type: apikey
        apikey: BASE64IDKEY==
    
      # options: <k:v> are appended as DSN query parameters when present
  3. Alias on /metrics
    Requests that already contain target= are internally forwarded to the probe handler, so users may keep path: /metrics in ServiceMonitors if they wish.

  4. Per-module API key support
    type: apikey injects the Authorization: ApiKey … header for that probe only.

  5. Docs & examples

    • README: updated multi-target section with full table of supported auth types.
    • examples/auth_modules.yml shows userpass, apikey.
  6. Unit tests

    • config loader round-trip.
    • Validation edge cases for validateProbeParams.

───────────────────────────────
Compatibility
───────────────────────────────

  • Single-target usage unchanged/metrics without target= still returns the startup cluster.
  • All existing CLI flags (TLS, indices, AWS signing, …) still apply.
  • No exported metrics renamed.

───────────────────────────────
Prometheus sample config
───────────────────────────────

- job_name: es-multi
  metrics_path: /probe
  params:
    auth_module: [prod_basic]
  static_configs:
    - targets:
        - https://es-prod:9200
    - targets:
        - https://es-staging:9200
      labels:
        __param_auth_module: staging_key
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: exporter:9114

───────────────────────────────
Commit outline
───────────────────────────────

  1. refactor: probe handler, alias logic
  2. feat: YAML auth-module loader (userpass, apikey)
  3. docs: README + example file
  4. tests: config loader & validation
  5. CHANGELOG entry

Feedback / suggestions welcome!

@pincher95 pincher95 force-pushed the add-multi-target-support branch 3 times, most recently from b3486c1 to 8bafbac Compare July 28, 2025 18:08
Copy link
Contributor

@sysadmind sysadmind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to spend more time with the code to provide all of my feedback, but want to provide what I have ready while I gave it the first pass.

@pincher95 pincher95 requested a review from sysadmind July 31, 2025 21:17
Copy link
Contributor

@sysadmind sysadmind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the guts of this are really solid. It did take me some time to give a proper review so once the changes are pushed up, I will need to allocate some time to re-review. Thanks for submitting this, I think it will be a good improvement.

@pincher95 pincher95 requested a review from sysadmind August 3, 2025 15:20
Copy link
Contributor

@sysadmind sysadmind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of the new files are also missing the copyright assignment/license header. You should be able to copy paste (make sure there are no dates) from any other go file. I don't love the extra requests for the two collectors but I think that can be cleaned up after #1052 is merged.

@pincher95 pincher95 requested a review from sysadmind August 10, 2025 14:15
Copy link
Contributor

@sysadmind sysadmind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The DCO check is also failing on at least one commit

pincher95 and others added 20 commits August 11, 2025 08:30
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
…'s empty and if so, don't parse it

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Update examples/auth_modules.yml
Fix main.go to apply userpass credentials only if the module type is explicitly set to userpass.

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Keep light-weight validation for the probe params during runtime
Add AWS SigV4 authentication module support

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Add TLS config validation
Update config test to include TLS config

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
…when no clusterinfo retriever is attached, avoiding the previous "unknown_cluster" label.

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
…sts to /probe.

Updated auth_modules.yml to include AWS SigV4 signing and mTLS support.

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
…theus-community#1065)

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.22.0 to 1.23.0.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.22.0...v1.23.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-version: 1.23.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Add tls auth type support in multi-target mode
Update README.md, examples/auth_modules.yml, tests

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
@pincher95 pincher95 force-pushed the add-multi-target-support branch from 955d9ee to bd1c0a8 Compare August 11, 2025 12:31
Signed-off-by: Yuri Tsuprun <51751791+pincher95@users.noreply.github.com>
@pincher95 pincher95 requested a review from sysadmind August 11, 2025 12:36
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Update roundtripper.go to use region from config or environment resolver if not provided in config file (AWS_REGION)
Update probe.go to accept module even if region omitted; environment resolver can provide it
Update config.go to use region as optional field
Update main.go to use region from config or environment resolver if not provided in config file (AWS_REGION) and update roundtripper.go to use region from config or environment resolver if not provided in config file (AWS_REGION)

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
@pincher95 pincher95 requested a review from sysadmind August 14, 2025 18:12
@sysadmind sysadmind merged commit ca4c313 into prometheus-community:master Aug 19, 2025
9 checks passed
@sysadmind
Copy link
Contributor

Thanks! Great stuff! I appreciate all of your hard work on this one to get it ready to ship.

pincher95 added a commit to pincher95/elasticsearch_exporter that referenced this pull request Nov 26, 2025
* Add multi-target support

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Update example-prometheus.yml

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Make `es.uri` optional by setting default to empty string check if it's empty and if so, don't parse it
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Update README.md

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Add sanity target scheme validation

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Change yaml package to go.yaml.in/yaml/v3

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Update yaml package to go.yaml.in/yaml/v3

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Update CHANGELOG.md

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Remove whitespaces from README.md

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Add testing for apikey authentication module
Update examples/auth_modules.yml
Fix main.go to apply userpass credentials only if the module type is explicitly set to userpass.

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Add Load-time validation for the auth module config file during startup
Keep light-weight validation for the probe params during runtime
Add AWS SigV4 authentication module support

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Expose error in the logger

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Add TLS config per target support
Add TLS config validation
Update config test to include TLS config

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Indices and Shards collectors now fetch cluster_name once from GET / when no clusterinfo retriever is attached, avoiding the previous "unknown_cluster" label.

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Removed the special-case logic that redirected /metrics?target= requests to /probe.
Updated auth_modules.yml to include AWS SigV4 signing and mTLS support.

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Add license headers to all new files

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Fixes for relative paths in multi-target mode

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Bump github.com/prometheus/client_golang from 1.22.0 to 1.23.0 (prometheus-community#1065)

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.22.0 to 1.23.0.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.22.0...v1.23.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-version: 1.23.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Add target schema validation, http/https only
Add tls auth type support in multi-target mode
Update README.md, examples/auth_modules.yml, tests

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Cleanup

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Fix tls auth type validation

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Remove aws.region validation

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Add temp file cleanup in config_test.go

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Add copyright header to config_test.go

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

* Add version metric to the per-probe registry
Update roundtripper.go to use region from config or environment resolver if not provided in config file (AWS_REGION)
Update probe.go to accept module even if region omitted; environment resolver can provide it
Update config.go to use region as optional field
Update main.go to use region from config or environment resolver if not provided in config file (AWS_REGION) and update roundtripper.go to use region from config or environment resolver if not provided in config file (AWS_REGION)

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>

---------

Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Yuri Tsuprun <51751791+pincher95@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: pincher95 <yuri.tsuprun@logz.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add multi-target support

2 participants