Skip to content

Commit 402c7fb

Browse files
leniatghleniatgh
authored
Apiscenario doc (#19896)
* update doc * gitignore * ci-fix * update quickstart * oav run -h * resize image * fix screenshot * generation * arm template * update reference * fix hashtag
1 parent 5a16258 commit 402c7fb

14 files changed

+627
-308
lines changed

‎.gitignore‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,10 @@ warnings.txt
110110
!.vscode/launch.json
111111
!.vscode/extensions.json
112112

113+
# API Test outputs
114+
.apitest
115+
113116
*.js
114117
*.d.ts
115118
*.js.map
116-
*.d.ts.map
119+
*.d.ts.map

‎documentation/api-scenario/how-to/QuickStart.md‎

Lines changed: 74 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -5,24 +5,27 @@
55
https://opensource.org/licenses/MIT
66
-->
77

8-
# API test quick start
8+
# Quick start with API Scenario test
99

1010
## Install
1111

12-
`oav` is an open-source powerful tool for swagger validation, example generation, and API testing. GitHub: https://github.com/Azure/oav.
12+
`oav` is an open-source powerful tool for swagger validation, API Scenario testing, and examples generation. GitHub: https://github.com/Azure/oav.
1313

1414
```sh
15-
npm install -g oav@latest
15+
npm install -g oav
1616
```
1717

1818
### OAV Features
1919

20-
- Very easy to use and run.
21-
- Support postman collection format. Debug easily.
22-
- Request response validation. `oav` implement a powerful validation algorithm and help developer to detect service issue in the early phase.
23-
- Validation result report. After each run API scenario, developer will get a validation report which contains detect issue in api test.
20+
- Very easy to use and run. It supports running API Scenario with ARM Dogfood/Canary/Production environments, and local environment as well.
21+
- Support Postman collection format. Debug easily.
22+
- Traffic schema validation and Azure API guidelines validation. `oav` implements a powerful validation algorithm and help developer to detect service issue in the early phase.
23+
- Generating high quality Swagger examples from API test traffic.
24+
- Validation result report. After each run API scenario, developer will get a validation report which contains detected issue in API test.
2425
- Integrate everywhere. Easily integrate with azure-pipeline, cloud-test.
2526

27+
See `oav run -h` to find all available options.
28+
2629
## Create AAD app
2730

2831
To run API test, first please prepare an AAD app which is used for provisioning Azure resource. Please grant subscription contributor permission to this AAD app.
@@ -31,7 +34,7 @@ For how to create AAD app, please follow this doc https://docs.microsoft.com/en-
3134

3235
## Authoring steps
3336

34-
We will write API scenario file for SignalR service as an example.
37+
We will write API scenario file for AppConfiguration service as an example.
3538

3639
#### 1. Write your first API scenario file
3740

@@ -44,68 +47,105 @@ Now write your basic API scenario. For more detail about API scenario file forma
4447

4548
```yaml
4649
# yaml-language-server: $schema=https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/documentation/api-scenario/references/v1.2/schema.json
50+
scope: ResourceGroup
51+
variables:
52+
configStoreName:
53+
type: string
54+
prefix: configstor
4755

56+
scenarios:
57+
- scenario: quickStart
58+
description: Quick start with AppConfiguration ConfigurationStores
59+
steps:
60+
- step: Operations_CheckNameAvailability
61+
exampleFile: ../examples/CheckNameAvailable.json
62+
- step: ConfigurationStores_Create
63+
exampleFile: ../examples/ConfigurationStoresCreate.json
64+
- step: ConfigurationStores_Get
65+
exampleFile: ../examples/ConfigurationStoresGet.json
66+
```
67+
68+
or use operation based step if Swagger examples are not ready or you want to create more scenarios without writing Swagger examples.
69+
70+
```yaml
71+
# yaml-language-server: $schema=https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/documentation/api-scenario/references/v1.2/schema.json
4872
scope: ResourceGroup
73+
variables:
74+
configStoreName:
75+
type: string
76+
prefix: configstor
77+
4978
scenarios:
5079
- scenario: quickStart
51-
description: Microsoft.SignalRService/signalR SignalR_CreateOrUpdate
80+
description: Quick start with AppConfiguration ConfigurationStores
5281
steps:
53-
- step: SignalR_CreateOrUpdate
54-
exampleFile: ../examples/SignalR_CreateOrUpdate.json
55-
- step: SignalR_Delete
56-
exampleFile: ../examples/SignalR_Delete.json
82+
- step: Operations_CheckNameAvailability
83+
operationId: Operations_CheckNameAvailability
84+
parameters:
85+
checkNameAvailabilityParameters:
86+
name: $(configStoreName)
87+
type: Microsoft.AppConfiguration/configurationStores
88+
- step: ConfigurationStores_Create
89+
operationId: ConfigurationStores_Create
90+
parameters:
91+
configStoreCreationParameters:
92+
location: $(location)
93+
sku:
94+
name: Standard
95+
tags:
96+
myTag: myTagValue
97+
- step: ConfigurationStores_Get
98+
operationId: ConfigurationStores_Get
5799
```
58100
59-
#### 2. create your env file
101+
#### 2. Create env file
60102
61103
The `env.json` file contains required API scenario variables such as, subscriptionId, AAD applicationId, AAD applicationSecret.
62104

63105
```json
64106
{
65107
"subscriptionId": "<my subscription id>",
66-
"location": "westus",
108+
"location": "westcentralus",
67109
"tenantId": "<AAD app tenantId>",
68110
"client_id": "<my add client_id>",
69111
"client_secret": "<my aad client_secret>"
70112
}
71113
```
72114

73-
#### 3. Run api test
115+
#### 3. Run API Scenario test
74116

75117
```sh
76-
oav run /home/user/azure-rest-api-specs/specification/signalr/resource-manager/Microsoft.SignalRService/preview/2020-07-01-preview/scenarios/signalR.yaml -e env.json
118+
oav run ~/workspace/azure-rest-api-specs/specification/appconfiguration/resource-manager/Microsoft.AppConfiguration/stable/2022-05-01/scenarios/quickstart.yaml --tag=package-2022-05-01 -e env.json --verbose
77119
```
78120

79-
#### 4. Debug with postman
121+
#### 4. Debug with Postman
80122

81-
Sometimes the command `oav run` may fail due to non 2xx HTTP status code. Now you need to debug the API scenario with postman.
123+
Sometimes the command `oav run` may fail due to non 2xx HTTP status code. Now you need to debug the API scenario with Postman.
82124

83-
When run `run`, it automatically generate postman collection and postman env in `generated/<providerNamespace>/<apiScenarioFile>/<runId>/<apiScenario>` folder. Here is the generated file folder structure. The `collection.json` and `env.json` is generated postman collection file and environment file. `202105120922-5c3x5` is current runId. For each run command it will generated unique runId.
125+
When run `run`, it automatically generate Postman collection and postman env in `.apitest/<apiScenarioFile>/<runId>/<scenario>` folder. Here is the output folder structure. The `collection.json` and `env.json` is generated postman collection file and environment file. `202207221820-cyq4mk` is the runId, generated uniquely for each run command.
84126

85127
```
86-
generated
87-
└── Microsoft.SignalRService
88-
└── 2020-07-01-preview
89-
└── signalR
90-
└── 202105120922-5c3x5
91-
├── signalR_0
92-
│ ├── collection.json
93-
│ └── env.json
94-
| |__ report.json
95-
└── signalR_0.json
128+
.apitest
129+
└── quickstart.yaml
130+
└── 202207221820-cyq4mk
131+
├── quickStart
132+
│ ├── collection.json
133+
│ ├── env.json
134+
│ └── report.json
135+
└── quickStart.json
96136
```
97137
98138
Postman is a widely used GUI API testing tool. And you could use Postman import the generated postman collection and env for your local debug.
99139
100140
![import-postman-collection](./import-postman-collection.png)
101141
102-
After you import postman collection, you will get such requests. Now you could debug API test with postman locally.
142+
After you import Postman collection, you will get such requests. Now you could debug API test with postman locally.
103143
104-
![postman-collection-signalr](./postman-collection-signalr.PNG)
144+
![postman-collection-list](./postman-collection-list.PNG)
105145
106-
#### 5. manual update example value
146+
#### 5. Manual update API Scenario or example
107147
108-
After debug with postman, you need to rewrite back all the updated values and run `oav run <api-scenario-file> -e <env.json>` again. The result should be successful.
148+
After debug with Postman, you need to rewrite back all the updated values and run `oav run <api-scenario-file> -e <env.json>` again. The result should be successful.
109149
110150
## Feedback
111151

‎documentation/api-scenario/how-to/apiScenarioWithARMTemplate.md‎

Lines changed: 0 additions & 105 deletions
This file was deleted.

‎documentation/api-scenario/how-to/folder-structure.png‎

100755100644
55.2 KB
Loading

‎documentation/api-scenario/how-to/genTestScenario.gif‎

-65.7 KB
Binary file not shown.

‎documentation/api-scenario/how-to/generateABasicApiScenario.md‎

Lines changed: 21 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -1,63 +1,36 @@
1-
# Generate a basic API scenario file
1+
# Generate a basic API Scenario file
22

3-
## Prerequisite
3+
In this section, we will show you how to generate a basic API scenario file from Swagger. This is useful for quickly generating a basic API scenario file as baseline, and you can improve the API Scenario (adjusting step orders and providing meaningful values) and make it runnable.
44

5-
We use `oav` tools to generate basic API scenario. `oav` analyze swagger file and use swagger example as API scenario steps. So first, you need to install the latest oav.
5+
## Prerequisites
66

7-
## Introduction
7+
1. Install [oav](https://www.npmjs.com/package/oav)
88

9-
`oav` support rule based API scenario file generation. We use this command to generate API scenario file.
10-
11-
`oav generate-api-scenario static --readme <readme> --tag <tag> --specs <specs> --rules <generated-rules>`
12-
13-
OR
14-
15-
`oav generate-api-scenario static --readme <readme> --tag <tag> --specs <specs> --dependency <dependency-path>`
16-
17-
- readme: swagger readme file.
18-
- tag: which tag to generate. oav will analyze swagger file under the tag and generate API scenario.
19-
- specs: one or more spec file paths. type: array.
20-
- dependency: The file path of the RESTler dependency. It cannot be used with `rules`.
21-
- rules: Currently support two types. `resource-put-delete`, `operations-list`. Default: `resource-put-delete`
22-
- `resource-put-delete`: generate resource put and delete API scenario.
23-
- `operations-list`: generate operations list API scenario. `operations-list` is the simplest API which must be defined in swagger.
24-
25-
Example:
9+
```bash
10+
npm i -g oav
11+
```
12+
2. Install [docker](https://docs.docker.com/get-docker/)
2613

27-
![](./genTestScenario.gif)
14+
## Steps
2815

29-
This command will load and analyze swagger and generate a basic API scenario file (`resource-put-delete`).
16+
1. Compile Swagger into dependencies.json with Restler.
3017

31-
Result: the output contains two files
18+
```bash
19+
docker run --rm -v $(pwd)/specification:/swagger -w /swagger/.restler_output mcr.microsoft.com/restlerfuzzer/restler:v8.5.0 dotnet /RESTler/restler/Restler.dll compile --api_spec /swagger/appconfiguration/resource-manager/Microsoft.AppConfiguration/stable/2022-05-01/appconfiguration.json
20+
```
3221

33-
- scenarios/signalR.yaml: The API scenario file.
34-
- readme.test.md: The entry for SDK test generation
22+
2. Generate a basic API Scenario file.
3523

36-
The generated API scenario file: The generated API scenario file contains two steps. Create signalR and delete it. It's a basic API scenario and developer can add more step based on the basic API scenario file.
24+
The generated API Scenario file will contain all the operations in the Swagger file, ordered by the dependencies. At each step, the minimum required parameters will be filled in.
3725

38-
```
39-
scope: ResourceGroup
40-
testScenarios:
41-
- description: Microsoft.SignalRService/signalR SignalR_CreateOrUpdate
42-
steps:
43-
- step: SignalR_CreateOrUpdate
44-
exampleFile: ../examples/SignalR_CreateOrUpdate.json
45-
- step: SignalR_Delete
46-
exampleFile: ../examples/SignalR_Delete.json
26+
```bash
27+
oav generate-api-scenario static --specs specification/appconfiguration/resource-manager/Microsoft.AppConfiguration/stable/2022-05-01/appconfiguration.json --dependency specification/.restler_output/Compile/dependencies.json -o specification/appconfiguration/resource-manager/Microsoft.AppConfiguration/stable/2022-05-01/scenarios
4728
```
4829

49-
If you pass rule option `operations-list`, you will get such API scenario file.
30+
As an alternative, if Swagger examples are ready, you can add `--useExample` parameter to generate the API scenario file based on Swagger examples:
5031

32+
```bash
33+
oav generate-api-scenario static --specs specification/appconfiguration/resource-manager/Microsoft.AppConfiguration/stable/2022-05-01/appconfiguration.json --dependency specification/.restler_output/Compile/dependencies.json -o specification/appconfiguration/resource-manager/Microsoft.AppConfiguration/stable/2022-05-01/scenarios --useExample
5134
```
52-
scope: ResourceGroup
53-
testScenarios:
54-
- description: operationsList
55-
steps:
56-
- step: operationsList
57-
exampleFile: ../examples/Operations_List.json
58-
59-
```
60-
61-
## Reference
6235

63-
- [oav](https://github.com/Azure/oav/tree/develop)
36+
Next you can run the API scenario file with `oav run`. See how in [QuickStart](./QuickStart.md).

0 commit comments

Comments
 (0)