`ManagedStreamingIngestClient` not handling `KustoThrottlingError`'s

[laegsgaardTroels]

Problem description

When using the ManagedStreamingIngestClient I've noticed that I sometimes get a KustoThrottlingError.

It is mentioned that this is handled in the docs, see:

Transient failures, for example throttling, are retried three times, then moved to queued ingestion.

I believe an except on KustoThrottlingError is needed here managed_streaming_ingest_client.py, line 97.

Output of pip freeze

azure-kusto-data==4.6.3
azure-kusto-ingest==4.6.3

[yogilad]

@laegsgaardTroels, can you share the exception you get?

[laegsgaardTroels]

@yogilad, thanks for getting back.

Below you have the stacktrace:

File "/usr/local/lib/python3.11/site-packages/azure/kusto/ingest/base_ingest_client.py", line 128, in ingest_from_dataframe
    return self.ingest_from_file(temp_file_path, ingestion_properties)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/kusto/ingest/managed_streaming_ingest_client.py", line 75, in ingest_from_file
    return self.ingest_from_stream(stream_descriptor, ingestion_properties)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/kusto/ingest/managed_streaming_ingest_client.py", line 93, in ingest_from_stream
    res = self._stream_with_retries(length, stream_descriptor, ingestion_properties)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/kusto/ingest/managed_streaming_ingest_client.py", line 142, in _stream_with_retries
    for attempt in Retrying(stop=stop_after_attempt(self._num_of_attempts), wait=wait_random_exponential(max=self._max_seconds_per_retry), reraise=True):
  File "/usr/local/lib/python3.11/site-packages/tenacity/__init__.py", line 445, in __iter__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tenacity/__init__.py", line 378, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tenacity/__init__.py", line 420, in exc_check
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tenacity/__init__.py", line 187, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.11/site-packages/azure/kusto/ingest/managed_streaming_ingest_client.py", line 151, in _stream_with_retries
    return MonitoredActivity.invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/kusto/data/_telemetry.py", line 116, in invoke
    return span()
           ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/kusto/ingest/managed_streaming_ingest_client.py", line 148, in <lambda>
    invoker = lambda: self.streaming_client._ingest_from_stream_with_client_request_id(descriptor, props, client_request_id)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/kusto/ingest/streaming_ingest_client.py", line 84, in _ingest_from_stream_with_client_request_id
    self._kusto_client.execute_streaming_ingest(
  File "/usr/local/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/kusto/data/client.py", line 271, in execute_streaming_ingest
    self._execute(endpoint, request, properties)
  File "/usr/local/lib/python3.11/site-packages/azure/kusto/data/client.py", line 362, in _execute
    raise self._handle_http_error(e, endpoint, request.payload, response, response.status_code, response_json, response.text)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/kusto/data/client_base.py", line 109, in _handle_http_error
    raise KustoThrottlingError("The request was throttled by the server.", response) from exception
azure.kusto.data.exceptions.KustoThrottlingError: ('The request was throttled by the server.', <Response [429]>)

[yogilad]

Yes, that makes sense.
Thanks for pointing this out.