Skip to content

Commit 77b3e8f

Browse files
authored
[SchemaRegistry] perfstress tests (Azure#22285)
* initial perfstress commit * fix * undo sample change * nit sample fix * adams comments * add num-schemas arg
1 parent 2fce666 commit 77b3e8f

File tree

6 files changed

+243
-0
lines changed

6 files changed

+243
-0
lines changed
Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# SchemaRegistry Performance Tests
2+
3+
In order to run the performance tests, the `azure-devtools` package must be installed. This is done as part of the `dev_requirements`.
4+
Start by creating a new virtual environment for your perf tests. This will need to be a Python 3 environment, preferably >=3.7.
5+
6+
### Setup for test resources
7+
8+
These tests will run against a pre-configured SchemaRegistry. The following environment variable will need to be set for the tests to access the live resources:
9+
```
10+
SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE=<the connection string of a Schema Registry.>
11+
SCHEMAREGISTRY_GROUP=<a schema group in a Schema Registry.>
12+
```
13+
14+
### Setup for perf test runs
15+
16+
```cmd
17+
(env) ~/azure-schemaregistry> pip install -r dev_requirements.txt
18+
(env) ~/azure-schemaregistry> pip install -e .
19+
```
20+
21+
## Test commands
22+
23+
When `azure-devtools` is installed, you will have access to the `perfstress` command line tool, which will scan the current module for runable perf tests. Only a specific test can be run at a time (i.e. there is no "run all" feature).
24+
25+
```cmd
26+
(env) ~/azure-schemaregistry> cd tests
27+
(env) ~/azure-schemaregistry/tests> perfstress
28+
```
29+
Using the `perfstress` command alone will list the available perf tests found.
30+
31+
### Common perf command line options
32+
These options are available for all perf tests:
33+
- `--duration=10` Number of seconds to run as many operations (the "run" function) as possible. Default is 10.
34+
- `--iterations=1` Number of test iterations to run. Default is 1.
35+
- `--parallel=1` Number of tests to run in parallel. Default is 1.
36+
- `--warm-up=5` Number of seconds to spend warming up the connection before measuring begins. Default is 5.
37+
- `--sync` Whether to run the tests in sync or async. Default is False (async).
38+
- `--no-cleanup` Whether to keep newly created resources after test run. Default is False (resources will be deleted).
39+
40+
### Schema Registry command line options
41+
The options are available for all SR perf tests:
42+
- `--schema-size=150` Number of bytes each schema contains, rounded down to nearest multiple of 50. Default is 150.
43+
- `--num-schemas` Number of schemas to register/get by ID/get properties for. Default is 10. May result in 'Forbidden' Exception for `RegisterSchemaTest` operation, if reached the limit of schemas allowed for Schema Registry tier.
44+
45+
### T2 Tests
46+
The tests currently written for the T2 SDK:
47+
- `RegisterSchemaTest` Registers `num-schemas` of size `schema-size` per run.
48+
- `GetSchemaByIdTest` Gets a registered schema of `schema-size` size `num-schemas` number of times.
49+
- `ReceiveEventBatchTest` Gets properties of a registered schema of `schema-size` size `num-schemas` number of times.
50+
51+
## Example command
52+
```cmd
53+
(env) ~/azure-schemaregistry/tests> perfstress GetSchemaByIdTest --parallel=2 --duration=100 --schema-size=500 --num-schemas=100
54+
```
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# --------------------------------------------------------------------------------------------
2+
# Copyright (c) Microsoft Corporation. All rights reserved.
3+
# Licensed under the MIT License. See License.txt in the project root for license information.
4+
# --------------------------------------------------------------------------------------------
Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
# --------------------------------------------------------------------------------------------
2+
# Copyright (c) Microsoft Corporation. All rights reserved.
3+
# Licensed under the MIT License. See License.txt in the project root for license information.
4+
# --------------------------------------------------------------------------------------------
5+
6+
import json
7+
import math
8+
import sys
9+
10+
from azure_devtools.perfstress_tests import PerfStressTest
11+
from azure.identity import DefaultAzureCredential
12+
from azure.identity.aio import DefaultAzureCredential as AsyncDefaultAzureCredential
13+
from azure.schemaregistry import SchemaRegistryClient
14+
from azure.schemaregistry.aio import SchemaRegistryClient as AsyncSchemaRegistryClient
15+
16+
17+
class _SchemaRegistryTest(PerfStressTest):
18+
def __init__(self, arguments):
19+
super().__init__(arguments)
20+
21+
self.fully_qualified_namespace = self.get_from_env(
22+
"SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE"
23+
)
24+
self.group_name = self.get_from_env("SCHEMAREGISTRY_GROUP")
25+
self.name = "your-schema-name"
26+
self.format = "Avro"
27+
self.definition = self._create_schema_definition()
28+
29+
def _create_schema_definition(self):
30+
schema_size = self.args.schema_size
31+
32+
fields = []
33+
schema = {
34+
"type": "record",
35+
"name": "example.User",
36+
"fields": fields,
37+
}
38+
39+
# 100 bytes
40+
schema_no_fields_size = sys.getsizeof(json.dumps(schema, separators=(",", ":")))
41+
fields.append({"name": "favor_number00000", "type": ["int", "null"]})
42+
# each additional field is 50 bytes
43+
schema_one_field_size = sys.getsizeof(json.dumps(schema, separators=(",", ":")))
44+
field_size = schema_one_field_size - schema_no_fields_size
45+
46+
# calculate number of fields to add to get args.schema_size rounded down to nearest 50 multiple
47+
num_fields = math.floor((schema_size - schema_no_fields_size) / field_size)
48+
49+
for i in range(1, num_fields):
50+
num_idx = f"{i:05d}"
51+
fields.append(
52+
{"name": f"favo_number{num_idx}", "type": ["int", "null"]},
53+
)
54+
definition = json.dumps(schema, separators=(",", ":"))
55+
return definition
56+
57+
@staticmethod
58+
def add_arguments(parser):
59+
super(_SchemaRegistryTest, _SchemaRegistryTest).add_arguments(parser)
60+
parser.add_argument(
61+
"--schema-size",
62+
nargs="?",
63+
type=int,
64+
help="Size of a single schema. Max 1000000 bytes. Defaults to 150 bytes",
65+
default=150,
66+
)
67+
parser.add_argument(
68+
"--num-schemas",
69+
nargs="?",
70+
type=int,
71+
help="""Number of schemas to register/get by ID/get properties for. Default is 10.
72+
May result in 'Forbidden' Exception for `RegisterSchemaTest` operation, if reached
73+
the limit of schemas allowed for Schema Registry tier.""",
74+
default=10,
75+
)
76+
77+
78+
class _RegisterTest(_SchemaRegistryTest):
79+
def __init__(self, arguments):
80+
super().__init__(arguments)
81+
self.sync_credential = DefaultAzureCredential()
82+
self.sync_client = SchemaRegistryClient(
83+
fully_qualified_namespace=self.fully_qualified_namespace,
84+
credential=self.sync_credential,
85+
)
86+
self.async_credential = AsyncDefaultAzureCredential()
87+
self.async_client = AsyncSchemaRegistryClient(
88+
fully_qualified_namespace=self.fully_qualified_namespace,
89+
credential=self.async_credential,
90+
)
91+
92+
async def global_setup(self):
93+
await super().global_setup()
94+
95+
async def close(self):
96+
self.sync_client.close()
97+
self.sync_credential.close()
98+
await self.async_client.close()
99+
await self.async_credential.close()
100+
await super().close()
101+
102+
103+
class _GetSchemaTest(_SchemaRegistryTest):
104+
def __init__(self, arguments):
105+
super().__init__(arguments)
106+
self.sync_credential = DefaultAzureCredential()
107+
self.sync_client = SchemaRegistryClient(
108+
fully_qualified_namespace=self.fully_qualified_namespace,
109+
credential=self.sync_credential,
110+
)
111+
self.async_credential = AsyncDefaultAzureCredential()
112+
self.async_client = AsyncSchemaRegistryClient(
113+
fully_qualified_namespace=self.fully_qualified_namespace,
114+
credential=self.async_credential,
115+
)
116+
self.schema_id = self._preregister_schema()
117+
118+
def _preregister_schema(self):
119+
with self.sync_client as client:
120+
schema_properties = client.register_schema(
121+
self.group_name, self.name, self.definition, self.format
122+
)
123+
return schema_properties.id
124+
125+
async def global_setup(self):
126+
await super().global_setup()
127+
128+
async def close(self):
129+
self.sync_client.close()
130+
self.sync_credential.close()
131+
await self.async_client.close()
132+
await self.async_credential.close()
133+
await super().close()
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# --------------------------------------------------------------------------------------------
2+
# Copyright (c) Microsoft Corporation. All rights reserved.
3+
# Licensed under the MIT License. See License.txt in the project root for license information.
4+
# --------------------------------------------------------------------------------------------
5+
6+
from ._test_base import _GetSchemaTest
7+
8+
9+
class GetSchemaByIdTest(_GetSchemaTest):
10+
def run_sync(self):
11+
for _ in range(self.args.num_schemas):
12+
self.sync_client.get_schema(self.schema_id)
13+
14+
async def run_async(self):
15+
for _ in range(self.args.num_schemas):
16+
await self.async_client.get_schema(self.schema_id)
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# --------------------------------------------------------------------------------------------
2+
# Copyright (c) Microsoft Corporation. All rights reserved.
3+
# Licensed under the MIT License. See License.txt in the project root for license information.
4+
# --------------------------------------------------------------------------------------------
5+
6+
from ._test_base import _GetSchemaTest
7+
8+
class GetSchemaPropertiesTest(_GetSchemaTest):
9+
def run_sync(self):
10+
for _ in range(self.args.num_schemas):
11+
self.sync_client.get_schema_properties(self.group_name, self.name, self.definition, self.format)
12+
13+
async def run_async(self):
14+
for _ in range(self.args.num_schemas):
15+
await self.async_client.get_schema_properties(self.group_name, self.name, self.definition, self.format)
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# --------------------------------------------------------------------------------------------
2+
# Copyright (c) Microsoft Corporation. All rights reserved.
3+
# Licensed under the MIT License. See License.txt in the project root for license information.
4+
# --------------------------------------------------------------------------------------------
5+
6+
import uuid
7+
from ._test_base import _RegisterTest
8+
9+
10+
class RegisterSchemaTest(_RegisterTest):
11+
def run_sync(self):
12+
for _ in range(self.args.num_schemas):
13+
self.sync_client.register_schema(
14+
self.group_name, self.name + str(uuid.uuid4()), self.definition, self.format
15+
)
16+
17+
async def run_async(self):
18+
for _ in range(self.args.num_schemas):
19+
await self.async_client.register_schema(
20+
self.group_name, self.name + str(uuid.uuid4()), self.definition, self.format
21+
)

0 commit comments

Comments
 (0)