Skip to content

Commit 5806c23

Browse files
m-reddingjsquirebenbp
authored
Eventhub stress roles (Azure#28930)
* stress test initial setup * initial setup of publish test after refactoring/app insights integration * added basic buffered test * deployment infrastructure update * updated buffered producer test * buffered producer test cleanup * buffered producer test * buffered producer test * Adding more scenarios and switching to event producer test * restructuring metrics * more metrics updates * README updates * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/README.md Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Infrastructure/EnvironmentReader.cs Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * initial refactoring * deploy yaml file updates * readme updates * Delete stress-test-resources.json * dockerfile updates * Delete stress-test-resources.json * comment and main updates * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/README.md Co-authored-by: Ben Broderick Phillips <ben@benbp.net> * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Infrastructure/Publisher.cs Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Configurations/BufferedProducerTestConfig.cs Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Configurations/BufferedProducerTestConfig.cs Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * feedback updates * Added Azure Event Listener * buffered producer options * typo * Event Hubs Stress Test Onboarding * Event Hubs Stress Test Onboarding * dockerfile * roles testing * role updates * documentation * more test updates * processor test updates * package update * added burst test * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Infrastructure/EnvironmentVariables.cs Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * name change * role explanation * feedback updates * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Roles/BufferedPublisher.cs Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Program.cs Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Program.cs Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Program.cs Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Program.cs Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Roles/Publisher.cs Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Roles/Processor.cs Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Roles/Processor.cs Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Roles/Processor.cs Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * Update sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Roles/Publisher.cs Co-authored-by: Jesse Squire <jesse.squire@gmail.com> * feedback changes * feedback updates * Update Packages.Data.props * Update Packages.Data.props * Update Packages.Data.props * blobs version * long test run Co-authored-by: Jesse Squire <jesse.squire@gmail.com> Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
1 parent 0d251f5 commit 5806c23

28 files changed

+1709
-873
lines changed

‎eng/Packages.Data.props‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -180,7 +180,7 @@
180180
<PackageReference Update="Azure.ResourceManager.Storage" Version="1.0.0-beta.8" />
181181
<PackageReference Update="Azure.Search.Documents" Version="11.2.0" />
182182
<PackageReference Update="Azure.Security.KeyVault.Secrets" Version="4.2.0-beta.4" />
183-
<PackageReference Update="Azure.Storage.Blobs" Version="12.10.0" />
183+
<PackageReference Update="Azure.Storage.Blobs" Version="12.12.0" />
184184
<PackageReference Update="Azure.Storage.Files.DataLake" Version="12.8.0" />
185185
<PackageReference Update="BenchmarkDotNet" Version="0.13.1" />
186186
<PackageReference Update="Castle.Core" Version="4.4.0" />

‎sdk/eventhub/Azure.Messaging.EventHubs/stress/Chart.lock‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
dependencies:
22
- name: stress-test-addons
33
repository: https://stresstestcharts.blob.core.windows.net/helm/
4-
version: 0.1.13
5-
digest: sha256:007ec9983233e0f8c8ad8aa7f1df5f57b1b4c127d223d59d233b3c5366bd5173
6-
generated: "2022-04-12T11:20:37.5250017-07:00"
4+
version: 0.1.16
5+
digest: sha256:75061792fdbebe57c664322579af6dc240ce8f464280ac5e192de72a29d4f63f
6+
generated: "2022-05-06T10:58:59.460663-07:00"

‎sdk/eventhub/Azure.Messaging.EventHubs/stress/Chart.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
apiVersion: v2
2-
name: eventhub-net-stress-test-2
2+
name: eventhub-net-stress
33
version: 0.1.2
44
description: Stress tests for Event Hubs for .NET
55

66
dependencies:
77
- name: stress-test-addons
8-
version: 0.1.13
8+
version: 0.1.16
99
repository: https://stresstestcharts.blob.core.windows.net/helm/
1010

1111
annotations:

‎sdk/eventhub/Azure.Messaging.EventHubs/stress/Dockerfile‎

Lines changed: 11 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,20 @@
1-
# right version of dotnet?
21
FROM mcr.microsoft.com/dotnet/sdk:6.0-cbl-mariner1.0 AS build-env
32

43
# Copy in engineering system needed to build
5-
COPY ./eng /app/eng
6-
COPY ./tools /app/tools
7-
COPY ./*.proj /app
8-
COPY ./Directory.Build.props /app
9-
COPY ./Directory.Build.targets /app
10-
COPY ./NuGet.Config /app
4+
COPY ./eng/ /app/eng/
5+
COPY ./tools/ /app/tools/
6+
COPY ./*.proj /app/
7+
COPY ./Directory.Build.props/ /app/
8+
COPY ./Directory.Build.targets/ /app/
9+
COPY ./NuGet.Config/ /app/
1110

1211
# Copy in Event Hubs Source Code
1312
COPY ./sdk/eventhub/Azure.Messaging.EventHubs/ /app/sdk/eventhub/Azure.Messaging.EventHubs/
14-
COPY ./sdk/eventhub/Azure.Messaging.EventHubs.Shared /app/sdk/eventhub/Azure.Messaging.EventHubs.Shared
13+
COPY ./sdk/eventhub/Azure.Messaging.EventHubs.Shared/ /app/sdk/eventhub/Azure.Messaging.EventHubs.Shared/
14+
COPY ./sdk/eventhub/Azure.Messaging.EventHubs.Processor/ /app/sdk/eventhub/Azure.Messaging.EventHubs.Processor/
1515

1616
# Copy in Core Source Code
17-
COPY ./sdk/core /app/sdk/core
17+
COPY ./sdk/core/ /app/sdk/core/
1818

1919
# Run restore and build, copy the .dll files over so they can used in the running container
2020
WORKDIR /app
@@ -23,12 +23,10 @@ RUN dotnet build './sdk/eventhub/Azure.Messaging.EventHubs/stress/src/' --config
2323
# Copy in the dll files to be ready to run
2424
FROM build-env as publish
2525
WORKDIR /app
26-
COPY --from=build-env /app/artifacts/bin/Azure.Messaging.EventHubs.Stress/Release/net6.0 /app/artifacts/bin/Azure.Messaging.EventHubs.Stress/Release/net6.0
27-
COPY --from=build-env /app/sdk/eventhub/Azure.Messaging.EventHubs.Shared /app/sdk/evethub/Azure.Messaging.EventHubs.Shared
28-
COPY --from=build-env /app/sdk/eventhub/Azure.Messaging.EventHubs/stress /app/sdk/eventhub/Azure.Messaging.EventHubs/stress
26+
COPY --from=build-env /app/artifacts/bin/Azure.Messaging.EventHubs.Stress/Release/net6.0/ /app/artifacts/bin/Azure.Messaging.EventHubs.Stress/Release/net6.0/
2927

3028
WORKDIR /app/artifacts/bin/Azure.Messaging.EventHubs.Stress/Release/net6.0
3129

3230
# The default is running just the "EventProducerTest"
33-
ENTRYPOINT ["dotnet Azure.Messaging.EventHubs.Stress.dll", "--tests"]
31+
ENTRYPOINT ["dotnet Azure.Messaging.EventHubs.Stress.dll", "--test"]
3432
CMD ["EventProducerTest"]

‎sdk/eventhub/Azure.Messaging.EventHubs/stress/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@ The scenarios in this directory provide a suite of stress tests that test the Ev
33

44
Test runs can call any of the following tests:
55
- "EventProducerTest"
6-
- "EventBufferedProducerTest"
6+
- "BufferedProducerTest"
77
- "BurstBufferedProducerTest"
8-
- "ConcurrentBufferedProducerTest"
8+
- "ProcessorTest"
99

1010
Or, you can run all tests with the "--all" flag.
1111

‎sdk/eventhub/Azure.Messaging.EventHubs/stress/charts/stress-test-addons-0.1.13.tgz‎

-171 KB
Binary file not shown.

‎sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Azure.Messaging.EventHubs.Stress.csproj‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616

1717
<ItemGroup>
1818
<ProjectReference Include="..\..\src\Azure.Messaging.EventHubs.csproj" />
19+
<ProjectReference Include="..\..\..\Azure.Messaging.EventHubs.Processor\src\Azure.Messaging.EventHubs.Processor.csproj" />
1920
</ItemGroup>
2021

2122
<!-- Import Azure.Core shared source -->

‎sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Configurations/BufferedProducerTestConfig.cs‎

Lines changed: 0 additions & 20 deletions
This file was deleted.

‎sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Configurations/BufferedPublisherConfiguration.cs‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
// Copyright (c) Microsoft Corporation. All rights reserved.
2+
// Licensed under the MIT License.
3+
4+
using System;
5+
using System.Collections.Generic;
6+
7+
namespace Azure.Messaging.EventHubs.Stress;
8+
9+
/// <summary>
10+
/// The set of configurations that can be specified when creating a <see cref="BufferedPublisher" />
11+
/// role.
12+
/// </summary>
13+
///
14+
internal class BufferedPublisherConfiguration
15+
{
16+
// Test Configuration Values
17+
18+
/// <summary>
19+
/// The number of events to generate for each call to
20+
/// <see cref="EventHubBufferedProducerClient.EnqueueEventsAsync(IEnumerable{EventData}, EnqueueEventOptions, CancellationToken)" />.
21+
/// </summary>
22+
///
23+
public int EventEnqueueListSize = 50;
24+
25+
/// <summary>
26+
/// The maximum size in bytes for the list of events to generate per call to
27+
/// <see cref="EventHubBufferedProducerClient.EnqueueEventsAsync(IEnumerable{EventData}, EnqueueEventOptions, CancellationToken)" />.
28+
/// </summary>
29+
///
30+
public long MaximumEventListSize = 1_000_000;
31+
32+
/// <summary>
33+
/// The number of concurrent sends going to the same <see cref="EventHubBufferedProducerClient" />.
34+
/// </summary>
35+
///
36+
public int ConcurrentSends = 5;
37+
38+
/// <summary>
39+
/// The minimum body size in bytes of events to generate when sending to the event hub.
40+
/// </summary>
41+
///
42+
public int PublishingBodyMinBytes = 100;
43+
44+
/// <summary>
45+
/// The maximum body size in bytes of events to generate when sending to the event hub.
46+
/// </summary>
47+
///
48+
public int PublishingBodyRegularMaxBytes = 262144;
49+
50+
/// <summary>
51+
/// The percentage of generated events for each send that have large bodies.
52+
/// </summary>
53+
///
54+
public int LargeMessageRandomFactorPercent = 50;
55+
56+
/// <summary>
57+
/// The amount of time to wait between enqueuing <see cref="EnqueueListSize"/> events.
58+
/// </summary>
59+
///
60+
public TimeSpan? ProducerPublishingDelay = TimeSpan.FromMilliseconds(4000);
61+
62+
// Buffered Producer Configuration Values
63+
64+
/// <summary>
65+
/// The value to use for <see cref="EventHubBufferedProducerClientOptions.MaximumWaitTime" />
66+
/// when configuring the <see cref="EventHubBufferedProducerClient" />.
67+
/// </summary>
68+
///
69+
public TimeSpan MaxWaitTime = TimeSpan.FromSeconds(5);
70+
71+
/// <summary>
72+
/// The value to use for <see cref="EventHubsRetryOptions.TryTimeout" /> when configuring
73+
/// the <see cref="EventHubBufferedProducerClient" />.
74+
/// </summary>
75+
///
76+
public TimeSpan SendTimeout = TimeSpan.FromMinutes(3);
77+
}

‎sdk/eventhub/Azure.Messaging.EventHubs/stress/src/Configurations/EventProducerTestConfig.cs‎

Lines changed: 0 additions & 13 deletions
This file was deleted.

0 commit comments

Comments
 (0)