feat: added backoff for job worker

[LennartKleymann]

closes #813

Feature: Configurable Exponential Backoff for Job Worker

This PR adds a configurable exponential backoff strategy for job polling retries in the C# client. This new feature helps prevent overwhelming the gateway and aligns the C# client with the behavior of the Java client.

Motivation

When the gateway is under heavy load and returns a RESOURCE_EXHAUSTED gRPC error, the current worker's aggressive retry behavior can create a "thundering herd" problem. By adding an exponential backoff, the worker can space out its retries, giving the gateway time to recover and improving overall system resilience.

Changes & Usage

• Adds IBackoffSupplier and IExponentialBackoffBuilder: These new APIs provide a fluent builder to configure the backoff policy.
• New Worker Builder API: Use the new BackoffSupplier() method on the job worker builder to supply a custom backoff strategy.
  • Example:

    var backoffSupplier = new ExponentialBackoffBuilder()
        .MinDelay(TimeSpan.FromMilliseconds(50))
        .MaxDelay(TimeSpan.FromSeconds(5))
        .BackoffFactor(1.6)
        .JitterFactor(0.1)
        .Build();
    
    client.NewWorker()
          .JobType("payment")
          .Handler(handler)
          .BackoffSupplier(backoffSupplier)
          .Open();

• Backoff Logic: The backoff is applied only on RESOURCE_EXHAUSTED errors and is reset to the initial polling interval after a successful job activation.

Backwards Compatibility

Existing worker configurations will continue to function without errors. However, the default polling behavior for these workers is now an exponential backoff with standard values, which differs from the previous fixed retry mechanism. The API remains backward compatible.

Testing

Unit tests have been added to verify the backoff behavior, including the handling of jitter and the monotonic increase of the delay.

[nloding]

cloned and tested locally, everything works as advertised! great work!

[ChrisKujawa]

Thanks @LennartKleymann, I like it 🙇🏼 Can you please resolve the merge conflicts and also apply the backoff supplier to the job streaming? I think then we can also remove the constants we had before (added here #812) wdyt?

After that we should be fine to merge

[ChrisKujawa]

👍🏼

[ChrisKujawa]

🔧 Could make a note that this implementation is based on the Java client version

[ChrisKujawa]

https://github.com/camunda/camunda/blob/5764a0a3e6c3d3253c5c9608bf4478f8e2281af7/clients/java-deprecated/src/main/java/io/camunda/zeebe/client/impl/worker/ExponentialBackoff.java#L31

[LennartKleymann]

I added summary and remarks to the ExponentialBackoffSupplier.cs

[ChrisKujawa]

🔧 we could build this once as constant to always have it available wdyt?

[LennartKleymann]

I moved initialization to the constructor to have one default supplier per JobWorker.
If you think of a static instance somewhere can you give me a hint where you would see it?

[LennartKleymann]

I changed in my commit to backoff now every RpcException in Stream and Poll Jobs

[LennartKleymann]

@ChrisKujawa can you review again?