diff --git a/CHANGELOG.md b/CHANGELOG.md index 8d79a899ba..1f4fbee53e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,9 +11,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## Unreleased +### Added + +- `opentelemetry-instrumentation-asgi`: Add exemplars for `http.server.request.duration` and `http.server.duration` metrics + ([#3739](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3739)) +- `opentelemetry-instrumentation-wsgi`: Add exemplars for `http.server.request.duration` and `http.server.duration` metrics + ([#3739](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3739)) + ## Version 1.39.0/0.60b0 (2025-12-03) -### Added +### Added - `opentelemetry-instrumentation-requests`, `opentelemetry-instrumentation-wsgi`, `opentelemetry-instrumentation-asgi` Detect synthetic sources on requests, ASGI, and WSGI. ([#3674](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3674)) @@ -57,10 +64,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ([#3882](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3882)) - `opentelemetry-instrumentation-aiohttp-server`: delay initialization of tracer, meter and excluded urls to instrumentation for testability ([#3836](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3836)) -- Replace Python 3.14-deprecated `asyncio.iscoroutinefunction` with `inspect.iscoroutinefunction`. +- Replace Python 3.14-deprecated `asyncio.iscoroutinefunction` with `inspect.iscoroutinefunction`. ([#3880](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3880)) - `opentelemetry-instrumentation-elasticsearch`: Enhance elasticsearch query body sanitization - ([#3919](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3919)) + ([#3919](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3919)) - `opentelemetry-instrumentation-pymongo`: Fix span error descriptions ([#3904](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3904)) - build: bump ruff to 0.14.1 @@ -69,7 +76,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ([#3941](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3941)) - `opentelemetry-instrumentation-pymongo`: Fix invalid mongodb collection attribute type ([#3942](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3942)) -- `opentelemetry-instrumentation-aiohttp-client`: Fix metric attribute leakage +- `opentelemetry-instrumentation-aiohttp-client`: Fix metric attribute leakage ([#3936](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3936)) - `opentelemetry-instrumentation-aiohttp-client`: Update instrumentor to respect suppressing http instrumentation ([#3957](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3957)) @@ -97,7 +104,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ([#3743](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3743)) - Add `rstcheck` to pre-commit to stop introducing invalid RST ([#3777](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3777)) -- `opentelemetry-exporter-credential-provider-gcp`: create this package which provides support for supplying your machine's Application Default +- `opentelemetry-exporter-credential-provider-gcp`: create this package which provides support for supplying your machine's Application Default Credentials (https://cloud.google.com/docs/authentication/application-default-credentials) to the OTLP Exporters created automatically by OpenTelemetry Python's auto instrumentation. These credentials authorize OTLP traces to be sent to `telemetry.googleapis.com`. [#3766](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3766). - `opentelemetry-instrumentation-psycopg`: Add missing parameter `capture_parameters` to instrumentor. ([#3676](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3676)) @@ -128,7 +135,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ([#3670](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3670)) - `opentelemetry-instrumentation-httpx`: fix missing metric response attributes when tracing is disabled ([#3615](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3615)) -- `opentelemetry-instrumentation-fastapi`: Don't pass bounded server_request_hook when using `FastAPIInstrumentor.instrument()` +- `opentelemetry-instrumentation-fastapi`: Don't pass bounded server_request_hook when using `FastAPIInstrumentor.instrument()` ([#3701](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3701)) ### Added @@ -139,7 +146,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ([#3666](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3666)) - `opentelemetry-sdk-extension-aws` Add AWS X-Ray Remote Sampler with initial Rules Poller implementation ([#3366](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3366)) -- `opentelemetry-instrumentation`: add support for `OTEL_PYTHON_AUTO_INSTRUMENTATION_EXPERIMENTAL_GEVENT_PATCH` to inform opentelemetry-instrument about gevent monkeypatching +- `opentelemetry-instrumentation`: add support for `OTEL_PYTHON_AUTO_INSTRUMENTATION_EXPERIMENTAL_GEVENT_PATCH` to inform opentelemetry-instrument about gevent monkeypatching ([#3699](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3699)) - `opentelemetry-instrumentation`: botocore: Add support for AWS Step Functions semantic convention attributes ([#3737](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3737)) diff --git a/instrumentation/opentelemetry-instrumentation-asgi/src/opentelemetry/instrumentation/asgi/__init__.py b/instrumentation/opentelemetry-instrumentation-asgi/src/opentelemetry/instrumentation/asgi/__init__.py index fb809e6836..0fe1807526 100644 --- a/instrumentation/opentelemetry-instrumentation-asgi/src/opentelemetry/instrumentation/asgi/__init__.py +++ b/instrumentation/opentelemetry-instrumentation-asgi/src/opentelemetry/instrumentation/asgi/__init__.py @@ -824,13 +824,18 @@ async def __call__( duration_attrs_new = _parse_duration_attrs( attributes, _StabilityMode.HTTP ) + span_ctx = set_span_in_context(span) if self.duration_histogram_old: self.duration_histogram_old.record( - max(round(duration_s * 1000), 0), duration_attrs_old + max(round(duration_s * 1000), 0), + duration_attrs_old, + context=span_ctx, ) if self.duration_histogram_new: self.duration_histogram_new.record( - max(duration_s, 0), duration_attrs_new + max(duration_s, 0), + duration_attrs_new, + context=span_ctx, ) self.active_requests_counter.add( -1, active_requests_count_attrs @@ -838,11 +843,15 @@ async def __call__( if self.content_length_header: if self.server_response_size_histogram: self.server_response_size_histogram.record( - self.content_length_header, duration_attrs_old + self.content_length_header, + duration_attrs_old, + context=span_ctx, ) if self.server_response_body_size_histogram: self.server_response_body_size_histogram.record( - self.content_length_header, duration_attrs_new + self.content_length_header, + duration_attrs_new, + context=span_ctx, ) request_size = asgi_getter.get(scope, "content-length") @@ -854,11 +863,15 @@ async def __call__( else: if self.server_request_size_histogram: self.server_request_size_histogram.record( - request_size_amount, duration_attrs_old + request_size_amount, + duration_attrs_old, + context=span_ctx, ) if self.server_request_body_size_histogram: self.server_request_body_size_histogram.record( - request_size_amount, duration_attrs_new + request_size_amount, + duration_attrs_new, + context=span_ctx, ) if token: context.detach(token) diff --git a/instrumentation/opentelemetry-instrumentation-asgi/tests/test_asgi_middleware.py b/instrumentation/opentelemetry-instrumentation-asgi/tests/test_asgi_middleware.py index fdf328498b..8f7ebe4ff1 100644 --- a/instrumentation/opentelemetry-instrumentation-asgi/tests/test_asgi_middleware.py +++ b/instrumentation/opentelemetry-instrumentation-asgi/tests/test_asgi_middleware.py @@ -315,6 +315,49 @@ def setUp(self): self.env_patch.start() + # Helper to assert exemplars presence across specified histogram metric names. + def _assert_exemplars_present( + self, metric_names: set[str], context: str = "" + ): + metrics_list = self.memory_metrics_reader.get_metrics_data() + print(metrics_list) + metrics = [] + for resource_metric in ( + getattr(metrics_list, "resource_metrics", []) or [] + ): + for scope_metric in ( + getattr(resource_metric, "scope_metrics", []) or [] + ): + metrics.extend(getattr(scope_metric, "metrics", []) or []) + + found = {name: 0 for name in metric_names} + for metric in metrics: + if metric.name not in metric_names: + continue + for point in metric.data.data_points: + found[metric.name] += 1 + exemplars = getattr(point, "exemplars", None) + self.assertIsNotNone( + exemplars, + msg=f"Expected exemplars list attribute on histogram data point for {metric.name} ({context})", + ) + self.assertGreater( + len(exemplars or []), + 0, + msg=f"Expected at least one exemplar on histogram data point for {metric.name} ({context}) but none found.", + ) + for ex in exemplars or []: + if hasattr(ex, "span_id"): + self.assertNotEqual(ex.span_id, 0) + if hasattr(ex, "trace_id"): + self.assertNotEqual(ex.trace_id, 0) + for name, count in found.items(): + self.assertGreater( + count, + 0, + msg=f"Did not encounter any data points for metric {name} while checking exemplars ({context}).", + ) + # pylint: disable=too-many-locals def validate_outputs( self, @@ -935,6 +978,9 @@ async def test_user_agent_synthetic_test_detection(self): # Test each user agent case separately to avoid span accumulation for user_agent in test_cases: with self.subTest(user_agent=user_agent): + # Reinitialize test state for each iteration to avoid state pollution + self.setUp() + # Clear headers first self.scope["headers"] = [] @@ -958,9 +1004,6 @@ def update_expected_synthetic_test( outputs, modifiers=[update_expected_synthetic_test] ) - # Clear spans after each test case to prevent accumulation - self.memory_exporter.clear() - async def test_user_agent_non_synthetic(self): """Test that normal user agents are not marked as synthetic""" test_cases = [ @@ -973,6 +1016,9 @@ async def test_user_agent_non_synthetic(self): # Test each user agent case separately to avoid span accumulation for user_agent in test_cases: with self.subTest(user_agent=user_agent): + # Reinitialize test state for each iteration to avoid state pollution + self.setUp() + # Clear headers first self.scope["headers"] = [] @@ -996,9 +1042,6 @@ def update_expected_non_synthetic( outputs, modifiers=[update_expected_non_synthetic] ) - # Clear spans after each test case to prevent accumulation - self.memory_exporter.clear() - async def test_user_agent_synthetic_new_semconv(self): """Test synthetic user agent detection with new semantic conventions""" user_agent = b"Mozilla/5.0 (compatible; Googlebot/2.1)" @@ -1534,6 +1577,40 @@ async def test_asgi_metrics_both_semconv(self): ) self.assertTrue(number_data_point_seen and histogram_data_point_seen) + async def test_asgi_metrics_exemplars_expected_old_semconv(self): + """Failing test placeholder asserting exemplars should be present for duration histogram (old semconv).""" + app = otel_asgi.OpenTelemetryMiddleware(simple_asgi) + for _ in range(5): + self.seed_app(app) + await self.send_default_request() + await self.get_all_output() + self._assert_exemplars_present( + {"http.server.duration"}, context="old semconv" + ) + + async def test_asgi_metrics_exemplars_expected_new_semconv(self): + """Failing test placeholder asserting exemplars should be present for request duration histogram (new semconv).""" + app = otel_asgi.OpenTelemetryMiddleware(simple_asgi) + for _ in range(5): + self.seed_app(app) + await self.send_default_request() + await self.get_all_output() + self._assert_exemplars_present( + {"http.server.request.duration"}, context="new semconv" + ) + + async def test_asgi_metrics_exemplars_expected_both_semconv(self): + """Failing test placeholder asserting exemplars should be present for both duration histograms when both semconv modes enabled.""" + app = otel_asgi.OpenTelemetryMiddleware(simple_asgi) + for _ in range(5): + self.seed_app(app) + await self.send_default_request() + await self.get_all_output() + self._assert_exemplars_present( + {"http.server.duration", "http.server.request.duration"}, + context="both semconv", + ) + async def test_basic_metric_success(self): app = otel_asgi.OpenTelemetryMiddleware(simple_asgi) self.seed_app(app) @@ -1569,7 +1646,7 @@ async def test_basic_metric_success(self): self.assertEqual(point.count, 1) if metric.name == "http.server.duration": self.assertAlmostEqual( - duration, point.sum, delta=5 + duration, point.sum, delta=30 ) elif metric.name == "http.server.response.size": self.assertEqual(1024, point.sum) @@ -1754,7 +1831,7 @@ async def test_basic_metric_success_both_semconv(self): ) elif metric.name == "http.server.duration": self.assertAlmostEqual( - duration, point.sum, delta=5 + duration, point.sum, delta=30 ) self.assertDictEqual( expected_duration_attributes_old, diff --git a/instrumentation/opentelemetry-instrumentation-wsgi/src/opentelemetry/instrumentation/wsgi/__init__.py b/instrumentation/opentelemetry-instrumentation-wsgi/src/opentelemetry/instrumentation/wsgi/__init__.py index 1107287b68..6e08a1fa21 100644 --- a/instrumentation/opentelemetry-instrumentation-wsgi/src/opentelemetry/instrumentation/wsgi/__init__.py +++ b/instrumentation/opentelemetry-instrumentation-wsgi/src/opentelemetry/instrumentation/wsgi/__init__.py @@ -654,6 +654,7 @@ def _start_response( return _start_response # pylint: disable=too-many-branches + # pylint: disable=too-many-locals def __call__( self, environ: WSGIEnvironment, start_response: StartResponse ): @@ -718,19 +719,24 @@ def __call__( raise finally: duration_s = default_timer() - start + active_metric_ctx = trace.set_span_in_context(span) if self.duration_histogram_old: duration_attrs_old = _parse_duration_attrs( req_attrs, _StabilityMode.DEFAULT ) self.duration_histogram_old.record( - max(round(duration_s * 1000), 0), duration_attrs_old + max(round(duration_s * 1000), 0), + duration_attrs_old, + context=active_metric_ctx, ) if self.duration_histogram_new: duration_attrs_new = _parse_duration_attrs( req_attrs, _StabilityMode.HTTP ) self.duration_histogram_new.record( - max(duration_s, 0), duration_attrs_new + max(duration_s, 0), + duration_attrs_new, + context=active_metric_ctx, ) self.active_requests_counter.add(-1, active_requests_count_attrs) diff --git a/instrumentation/opentelemetry-instrumentation-wsgi/tests/test_wsgi_middleware.py b/instrumentation/opentelemetry-instrumentation-wsgi/tests/test_wsgi_middleware.py index bb6c3aca2f..637fa4c757 100644 --- a/instrumentation/opentelemetry-instrumentation-wsgi/tests/test_wsgi_middleware.py +++ b/instrumentation/opentelemetry-instrumentation-wsgi/tests/test_wsgi_middleware.py @@ -294,6 +294,41 @@ def validate_response( expected_attributes[HTTP_REQUEST_METHOD] = http_method self.assertEqual(span_list[0].attributes, expected_attributes) + # Helper modeled after ASGI test suite to assert presence of exemplars on histogram metrics + def _assert_exemplars_present(self, metric_names, context=""): + metrics_data = self.memory_metrics_reader.get_metrics_data() + self.assertTrue( + len(metrics_data.resource_metrics) > 0, + f"No resource metrics collected while checking exemplars ({context})", + ) + checked = set() + for resource_metric in metrics_data.resource_metrics: + for scope_metric in resource_metric.scope_metrics: + for metric in scope_metric.metrics: + if metric.name not in metric_names: + continue + checked.add(metric.name) + # Expect exactly one datapoint per histogram metric in these tests + data_points = list(metric.data.data_points) + self.assertGreater( + len(data_points), + 0, + f"No data points for {metric.name} while checking exemplars ({context})", + ) + for point in data_points: + if isinstance(point, HistogramDataPoint): + self.assertGreater( + len(point.exemplars), + 0, + f"Expected at least one exemplar on histogram data point for {metric.name} ({context}) but none found.", + ) + # Ensure we actually saw all targeted metrics + self.assertSetEqual( + set(metric_names), + checked, + f"Did not observe all targeted metrics when asserting exemplars ({context}). Expected {metric_names} got {checked}", + ) + def test_basic_wsgi_call(self): app = otel_wsgi.OpenTelemetryMiddleware(simple_wsgi) response = app(self.environ, self.start_response) @@ -418,6 +453,42 @@ def test_wsgi_metrics(self): ) self.assertTrue(number_data_point_seen and histogram_data_point_seen) + def test_wsgi_metrics_exemplars_expected_old_semconv(self): # type: ignore[func-returns-value] + """Failing test asserting exemplars should be present for duration histogram (old semconv).""" + app = otel_wsgi.OpenTelemetryMiddleware(simple_wsgi) + # generate several requests to increase chance of exemplar sampling + for _ in range(5): + response = app(self.environ, self.start_response) + # exhaust response iterable + for _ in response: + pass + self._assert_exemplars_present( + {"http.server.duration"}, context="old semconv" + ) + + def test_wsgi_metrics_exemplars_expected_new_semconv(self): # type: ignore[func-returns-value] + """Failing test asserting exemplars should be present for request duration histogram (new semconv).""" + app = otel_wsgi.OpenTelemetryMiddleware(simple_wsgi) + for _ in range(5): + response = app(self.environ, self.start_response) + for _ in response: + pass + self._assert_exemplars_present( + {"http.server.request.duration"}, context="new semconv" + ) + + def test_wsgi_metrics_exemplars_expected_both_semconv(self): # type: ignore[func-returns-value] + """Failing test asserting exemplars should be present for both duration histograms when both semconv modes enabled.""" + app = otel_wsgi.OpenTelemetryMiddleware(simple_wsgi) + for _ in range(5): + response = app(self.environ, self.start_response) + for _ in response: + pass + self._assert_exemplars_present( + {"http.server.duration", "http.server.request.duration"}, + context="both semconv", + ) + def test_wsgi_metrics_new_semconv(self): # pylint: disable=too-many-nested-blocks app = otel_wsgi.OpenTelemetryMiddleware(error_wsgi_unhandled)