Skip to content

Commit 91496d9

Browse files
HarshaNalluruHarshaNalluru
authored
[Perf Framework] Restructuring the perf test folders - Proposal 2 (Azure#13770)
## Description To allow testing the older versions of track-2 SDKs and for smoother automation with the pipelines, we are moving away from the in-place perf tests to independent projects under the service folders. This change doesn't affect how the perf tests are written. The main change here is that the track 2 perf tests are made into a project maintained by "rush" as opposed to in-place tests. Track 1 tests are not managed by "rush", it is an npm project. ### Old - `sdk/storage/storage-blob/test/perfstress/track-1/` - npm project (but imports the perf framework) - `sdk/storage/storage-blob/test/perfstress/track-2/` - depends on src code like the regular tests ### New - `sdk/storage/perf-tests/storage-blob-track-1/` - Independent npm project (but imports the perf framework) - `sdk/storage/perf-tests/storage-blob/`              - New project maintained through "rush" (depends on storage-blob on master) One key change is that the rush property - `"projectFolderMaxDepth"` is being changed from 3 to 4 to allow creating new projects at 4 levels down from the root. The following walk-through for executing perf tests uses storage-blob as an example. (Check Azure#13740 for proposal 1 - old.) ## Testing Track 1 - `rush update` - Navigate to `sdk\storage\perf-tests\storage-blob-track-1` - Run `npm run setup` (Builds the perf package and installs it for track-1 perf tests) - Add .env file as suggested in the readme - Run the tests as suggested by readme, example `npm run perf-test:node -- StorageBlobDownloadTest --warmup 2 --duration 7 --iterations 2 --parallel 2` ## Testing Track 2 - unpublished(current master) - `rush update` - Navigate to `sdk\storage\perf-tests\storage-blob` - `rush build -t perf-test-storage-blob` - Add .env file as suggested in the readme - Run the tests as suggested by readme, example `npm run perf-test:node -- StorageBlobDownloadTest --warmup 2 --duration 7 --iterations 2 --parallel 2` ## Testing Track 2 - older versions - Example: To test 12.2.0 - Update `"@azure/storage-blob"` version in `package.json` to `12.2.0` - Add a new exception in `common\config\rush\common-versions.json` under `allowedAlternativeVersions` - `"@azure/storage-blob": [..., "12.2.0"]` - `rush update` (generates a new pnpm-lock file) - Navigate to `sdk\storage\perf-tests\storage-blob` - `rush build -t perf-test-storage-blob` - Add .env file as suggested in the readme - Run the tests as suggested by readme, example `npm run perf-test:node -- StorageBlobDownloadTest --warmup 2 --duration 7 --iterations 2 --parallel 2`
1 parent dda2df0 commit 91496d9

35 files changed

+289
-93
lines changed

‎common/config/rush/common-versions.json‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,9 +41,10 @@
4141
* This design avoids unnecessary churn in this file.
4242
*/
4343
"allowedAlternativeVersions": {
44-
// The following is required for eventhubs-checkpointstore-blob to use the latest GA version
44+
// "^12.4.1" is required for eventhubs-checkpointstore-blob to use the latest GA version
4545
// when there is a new beta version which is being maintained in the repo.
46-
// Comment this out when the storage-blob releases a stable version.
46+
// Remove "^12.4.1" when the storage-blob releases a stable version.
47+
// Add a new entry in case a new version is being tested through the perf tests (Example: "12.2.0").
4748
"@azure/storage-blob": ["^12.4.1"],
4849
"@azure/ms-rest-js": ["^2.0.0"],
4950
/**

‎common/config/rush/pnpm-lock.yaml‎

Lines changed: 27 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎common/config/rush/version-policies.json‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,11 @@
33
* For full documentation, please see https://rushjs.io
44
*/
55

6-
/**
7-
* A list of version policy definitions. A "version policy" is a custom package versioning
8-
* strategy that affets "rush change", "rush version", and "rush publish". The strategy applies
9-
* to a set of projects that are specified using the "versionPolicyName" field in rush.json.
10-
*/
6+
/**
7+
* A list of version policy definitions. A "version policy" is a custom package versioning
8+
* strategy that affets "rush change", "rush version", and "rush publish". The strategy applies
9+
* to a set of projects that are specified using the "versionPolicyName" field in rush.json.
10+
*/
1111
[
1212
// {
1313
// /**
@@ -93,5 +93,9 @@
9393
{
9494
"definitionName": "individualVersion",
9595
"policyName": "utility"
96+
},
97+
{
98+
"definitionName": "individualVersion",
99+
"policyName": "test"
96100
}
97101
]

‎common/tools/perf-tests-track-1-setup.js‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
// Copyright (c) Microsoft Corporation.
2+
// Licensed under the MIT license.
3+
4+
/**
5+
* Track 1 perf tests are not maintained through rush in this repository.
6+
*
7+
* This helper script is meant to be used to "build" and "install"
8+
* the `@azure/test-utils-perfstress` package in the track-1 perf-test projects
9+
* so that the track 1 perf tests can leverage the perf framework.
10+
*
11+
* Additionally, this runs `npm install` in the track-1 perf test projects.
12+
*/
13+
const { spawnSync } = require("child_process");
14+
const path = require("path");
15+
16+
function spawn(command) {
17+
const ret = spawnSync(command, { shell: true, stdio: "inherit" });
18+
19+
if (ret.status !== 0) {
20+
throw new Error(`${command} failed with exit code ${ret.status}`);
21+
}
22+
}
23+
24+
const navigateToPerfStressFolder = `cd ${path.resolve(
25+
"..",
26+
"..",
27+
"..",
28+
"test-utils",
29+
"perfstress"
30+
)}`;
31+
const buildPerfStressPackage = `rush build -t test-utils-perfstress`;
32+
const rushxPack = `rushx pack`;
33+
34+
console.log(`\nGetting perfstress package...`);
35+
spawn(`${navigateToPerfStressFolder} && ${buildPerfStressPackage} && ${rushxPack}`);
36+
37+
console.log(`\nRunning "npm install"...`);
38+
spawn(`npm install`);

‎eng/.docsettings.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ omitted_paths:
2525
- sdk/schemaregistry/README.md
2626
- sdk/storage/*/test/README.md
2727
- sdk/storage/storage-internal-avro/*
28+
- sdk/storage/perf-tests/*/README.md
2829
- sdk/storage/*/test/perfstress/track-1/*
2930
- sdk/storage/*/test/perfstress/track-2/*
3031
- sdk/textanalytics/*/test/README.md

‎eng/tools/rush-runner.js‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -118,11 +118,23 @@ const getPackagesToBuild = (packageNames, packageGraph) => {
118118
};
119119

120120
const getPackageJsons = (searchDir) => {
121-
return fs
121+
// This gets all the directories with package.json at the `sdk/<service>/<service-sdk>` level excluding "arm-" packages
122+
const sdkDirectories = fs
122123
.readdirSync(searchDir)
123124
.filter((f) => !f.startsWith("arm-")) // exclude libraries starting with "arm-"
124-
.map((f) => path.join(searchDir, f, "package.json")) // turn potential directory names into package.json paths
125-
.filter((f) => fs.existsSync(f)); // only keep paths for files that actually exist
125+
.map((f) => path.join(searchDir, f, "package.json")); // turn potential directory names into package.json paths
126+
127+
// This gets all the directories with package.json at the `sdk/<service>/<service-sdk>/perf-tests` level excluding "-track-1" perf test packages
128+
let perfTestDirectories = [];
129+
const searchPerfTestDir = path.join(searchDir, "perf-tests");
130+
if (fs.existsSync(searchPerfTestDir)) {
131+
perfTestDirectories = fs
132+
.readdirSync(searchPerfTestDir)
133+
.filter((f) => !f.endsWith("-track-1")) // exclude libraries ending with "-track-1" (perf test projects)
134+
.map((f) => path.join(searchPerfTestDir, f, "package.json")); // turn potential directory names into package.json paths
135+
}
136+
137+
return sdkDirectories.concat(perfTestDirectories).filter((f) => fs.existsSync(f)); // only keep paths for files that actually exist
126138
};
127139

128140
const getServicePackages = (baseDir, serviceDirs) => {

‎rush.json‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,7 @@
110110
* and set projectFolderMaxDepth to a large number.
111111
*/
112112
"projectFolderMinDepth": 3,
113-
"projectFolderMaxDepth": 3,
113+
"projectFolderMaxDepth": 4,
114114
/**
115115
* This feature helps you to review and approve new packages before they are introduced
116116
* to your monorepo. For example, you may be concerned about licensing, code quality,
@@ -632,6 +632,11 @@
632632
"projectFolder": "sdk/digitaltwins/digital-twins-core",
633633
"versionPolicyName": "client"
634634
},
635+
{
636+
"packageName": "@azure-tests/perf-storage-blob",
637+
"projectFolder": "sdk/storage/perf-tests/storage-blob",
638+
"versionPolicyName": "test"
639+
},
635640
{
636641
"packageName": "@azure/mixedreality-authentication",
637642
"projectFolder": "sdk/mixedreality/mixedreality-authentication",

‎sdk/storage/perf-tests/storage-blob-track-1/README.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
### Guide
2+
3+
1. Navigate to `sdk\storage\perf-tests\storage-blob-track-1`
4+
2. Do `rush update`.
5+
3. Run `npm run setup`.
6+
4. Copy the `sample.env` file and name it as `.env`.
7+
5. Create a storage account and populate the `.env` file with `STORAGE_CONNECTION_STRING` variable.
8+
6. Run the tests as follows
9+
10+
- download
11+
- `npm run perf-test:node -- StorageBlobDownloadTest --warmup 2 --duration 7 --iterations 2 --parallel 2`
12+
- upload
13+
- `npm run perf-test:node -- StorageBlobUploadTest --warmup 2 --duration 7 --iterations 2 --parallel 2`
14+
- list blobs
15+
- `npm run perf-test:node -- StorageBlobListTest --warmup 2 --duration 7 --iterations 2 --parallel 2`

‎sdk/storage/perf-tests/storage-blob-track-1/package.json‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
{
2+
"name": "@azure-tests/perf-storage-blob-track-1",
3+
"version": "1.0.0",
4+
"description": "",
5+
"main": "",
6+
"keywords": [],
7+
"author": "",
8+
"license": "ISC",
9+
"devDependencies": {
10+
"@azure/storage-blob": "^10.5.0",
11+
"@azure/test-utils-perfstress": "file:../../../test-utils/perfstress/azure-test-utils-perfstress-1.0.0.tgz",
12+
"@types/uuid": "^8.0.0",
13+
"dotenv": "^8.2.0",
14+
"uuid": "^8.3.0"
15+
},
16+
"private": true,
17+
"scripts": {
18+
"audit": "node ../../../common/scripts/rush-audit.js && rimraf node_modules package-lock.json && npm i --package-lock-only 2>&1 && npm audit",
19+
"build": "tsc -p .",
20+
"build:samples": "echo skipped",
21+
"build:test": "echo skipped",
22+
"check-format": "prettier --list-different --config ../../../.prettierrc.json --ignore-path ../../../.prettierignore \"src/**/*.ts\" \"test/**/*.ts\" \"*.{js,json}\"",
23+
"clean": "rimraf dist dist-esm test-dist typings *.tgz *.log",
24+
"format": "prettier --write --config ../../../.prettierrc.json --ignore-path ../../../.prettierignore \"src/**/*.ts\" \"test/**/*.ts\" \"*.{js,json}\"",
25+
"integration-test:browser": "echo skipped",
26+
"integration-test:node": "echo skipped",
27+
"integration-test": "echo skipped",
28+
"lint:fix": "eslint package.json src test --ext .ts --fix --fix-type [problem,suggestion]",
29+
"lint": "eslint package.json src test --ext .ts -f html -o storage-blob-perf-test-lintReport.html || exit 0",
30+
"pack": "npm pack 2>&1",
31+
"perf-test:node": "ts-node test/index.spec.ts",
32+
"prebuild": "npm run clean",
33+
"setup": "node ../../../../common/tools/perf-tests-track-1-setup.js",
34+
"unit-test:browser": "echo skipped",
35+
"unit-test:node": "echo skipped",
36+
"unit-test": "echo skipped",
37+
"test:browser": "echo skipped",
38+
"test:node": "echo skipped",
39+
"test": "echo skipped"
40+
}
41+
}

‎sdk/storage/perf-tests/storage-blob-track-1/sample.env‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
STORAGE_CONNECTION_STRING=<connection-string>

0 commit comments

Comments
 (0)