gRPC exceptions when worker try to connect

[IlyaNavodkin]

Hi,

I'm using zeebe-client-csharp and noticed that when my workers are idle (i.e., there are no tasks to poll), the client floods my logs with gRPC exceptions like Status code: 'Cancelled' and gRPC call deadline exceeded.

warn: Grpc.Net.Client.Internal.GrpcCall[7]
      gRPC call deadline exceeded.
warn: Grpc.Net.Client.Internal.GrpcCall[7]
      gRPC call deadline exceeded.
warn: Grpc.Net.Client.Internal.GrpcCall[7]
      gRPC call deadline exceeded.
warn: Grpc.Net.Client.Internal.GrpcCall[7]
      gRPC call deadline exceeded.
warn: Grpc.Net.Client.Internal.GrpcCall[7]
      gRPC call deadline exceeded.
info: Grpc.Net.Client.Internal.GrpcCall[3]
      Call failed with gRPC error status. Status code: 'DeadlineExceeded', Message: ''.
info: Grpc.Net.Client.Internal.GrpcCall[3]
      Call failed with gRPC error status. Status code: 'DeadlineExceeded', Message: ''.
info: Grpc.Net.Client.Internal.GrpcCall[3]
      Call failed with gRPC error status. Status code: 'DeadlineExceeded', Message: ''.
info: Grpc.Net.Client.Internal.GrpcCall[3]
      Call failed with gRPC error status. Status code: 'DeadlineExceeded', Message: ''.
info: Grpc.Net.Client.Internal.GrpcCall[3]
      Call failed with gRPC error status. Status code: 'DeadlineExceeded', Message: ''.
info: Grpc.Net.Client.Internal.GrpcCall[3]
      Call failed with gRPC error status. Status code: 'Cancelled', Message: 'Error starting gRPC call. HttpRequestException: An error occurred while sending the request. HttpProtocolException: The HTTP/2 server reset the stream. HTTP/2 error code 'CANCEL' (0x8).'.
      System.Net.Http.HttpRequestException: An error occurred while sending the request.
       ---> System.Net.Http.HttpProtocolException: The HTTP/2 server reset the stream. HTTP/2 error code 'CANCEL' (0x8).
         at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
         at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
         at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
         at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
         at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.HttpMessageInvoker.<SendAsync>g__SendAsyncWithTelemetry|6_0(HttpMessageHandler handler, HttpRequestMessage request, CancellationToken cancellationToken)
         at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)

My code (use hosted service in ASP net core):

builder.Services.AddHostedService<CammundaWorker>(); /// in Programm.cs
.....

public class CammundaWorker : IHostedService
{
    public CammundaWorker(ILogger<CammundaWorker> logger, IZeebeClient zeebeClient)
    {
        _zeebeClient = zeebeClient;
        _logger = logger;
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        var topology = await _zeebeClient.TopologyRequest()
            .Send();

        foreach (var workerName in workersNames)
        {
            var worker = _zeebeClient.NewWorker()
               .JobType(workerName)
               .Handler(HandleCommon)
               .Name(workerName)
               .MaxJobsActive(1)
               .PollInterval(TimeSpan.FromSeconds(5))
               .Timeout(TimeSpan.FromSeconds(70))
               .AutoCompletion()
               .Open();

            _logger.LogInformation("Started worker: {WorkerName}", workerName);

            _workers.Add(worker);
        }

        _logger.LogInformation("All workers started successfully");

        await Task.CompletedTask;
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        _zeebeClient.Dispose();

        _logger.LogInformation("Stopped workers is successfully");

        return Task.CompletedTask;
    }
}

How did you plan for using workers in ASP.NET Core or other applications to prevent spam from these exceptions? Besides configuring log sensitivity (I sometimes care about exceptions that aren't from the Zeebee client).

Why are there constant exceptions and errors? Is this because each worker is pooled to the server by design, or am I doing something wrong? If so, is there a feature to intercept and ignore these exceptions? If not, why is this designed this way?

When the process is created, the workers run normally, and then the exceptions start again.

Thanks in advance!

[xlegalles]

Totally agree. And because of that, we have disabled all grpc.net.* logs by default. Of course this is not the ideal solution...