You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: documentation/swagger-accuracy-report.md
+46-30Lines changed: 46 additions & 30 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,61 +4,77 @@ The purpose of the Service Contract (Swagger) Accuracy Report is to ensure that
4
4
5
5
Our tooling ([RESTler](https://github.com/microsoft/restler-fuzzer) and [OAV](https://github.com/azure/oav)) that produces this report tests all service operations and compares the actual requests/responses against the contract for accuracy.
6
6
7
-
***
7
+
---
8
+
8
9
**IMPORTANT: _All operations must be tested and pass before the PR contract can be merged._**
9
-
***
10
+
11
+
---
10
12
11
13
For any `Failed Operations`, use the error information (and links provided) to either fix the contract or fix the service behavior. Fix failed operations first as this may enable our tooling to test some of the currently untested operations.
12
14
13
15
For any `Untested Operations`, you must modify the RESTler configuration files to improve API coverage.
14
16
15
-
1. Download the RESTler configuration files
16
-
- ...
17
+
## Steps to improve API coverage
18
+
19
+
### 1. Download the RESTler configuration files
20
+
21
+
In swagger PR pipeline, traffic validation pipeline will generate RESTler configuration files as artifacts. You could click the **pipeline artifact** to download the configuration files.
The red block in the above image is the generated restler configuration files and swaggerAccuracyReport. After downloading the configuration files, you could modify the configuration files and run restler locally to improve API coverage.
30
+
17
31
<!--
18
32
TODO: the RESTler config files do not include the authentication script. How to set up authentication / where to get the script needs to be documented for a local setup.
19
33
-->
20
34
21
-
2. Modify the configuration files to successfully execute more operations
35
+
### 2. Modify the configuration files to successfully execute more operations
22
36
23
-
In this CI pipeline, RESTler is running in [Test](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/Testing.md) mode,
24
-
where it tries to execute each operation in the Swagger specification successfully once.
25
-
When an operation has dependencies (e.g. parent resources),
26
-
RESTler tries to infer these dependencies and create the required parent resource.
27
-
To learn more about how RESTler works, please refer to the [README](https://github.com/microsoft/restler-fuzzer).
37
+
In this CI pipeline, RESTler is running in [Test](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/Testing.md) mode,
38
+
where it tries to execute each operation in the Swagger specification successfully once.
39
+
When an operation has dependencies (e.g. parent resources),
40
+
RESTler tries to infer these dependencies and create the required parent resource.
41
+
To learn more about how RESTler works, please refer to the [README](https://github.com/microsoft/restler-fuzzer).
28
42
29
-
You may either debug your changes locally by downloading RESTler, or for simpler fixes re-submit changes directly to the pipeline (in the latter case, refer to Step 3).
30
-
The advantage of a local setup is that you will get more familiar with RESTler, and can compare your known-good local setup
31
-
for testing the service with the setup in the CI pipeline to diagnose why a pipeline failure occurred.
43
+
You may either debug your changes locally by downloading RESTler, or for simpler fixes re-submit changes directly to the pipeline (in the latter case, refer to Step 3).
44
+
The advantage of a local setup is that you will get more familiar with RESTler, and can compare your known-good local setup
45
+
for testing the service with the setup in the CI pipeline to diagnose why a pipeline failure occurred.
32
46
33
-
<b>To run RESTler locally:</b>
47
+
<b>To run RESTler locally:</b>
34
48
35
-
### A. Download RESTler
49
+
**A. Download RESTler**
36
50
37
-
See the [README](https://github.com/microsoft/restler-fuzzer) for instructions.
51
+
See the [README](https://github.com/microsoft/restler-fuzzer) for instructions.
38
52
39
-
### B. Go through each failed operation in the RESTler logs and fix the configuration to make it succeed
53
+
**B. Go through each failed operation in the RESTler logs and fix the configuration to make it succeed**
40
54
41
-
There are 3 configuration files that you may need to modify:
55
+
There are 3 configuration files that you may need to modify:
42
56
43
-
- Dictionary (```dict.json```) - specify "magic constants" to assign to specific parameters. See [Dictionary](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/FuzzingDictionary.md).
44
-
- Annotation file (```annotations.json```) - specify dependencies between requests. See [Annotations](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/Annotations.md).
- Dictionary (`dict.json`) - specify "magic constants" to assign to specific parameters. See [Dictionary](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/FuzzingDictionary.md).
58
+
- Annotation file (`annotations.json`) - specify dependencies between requests. See [Annotations](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/Annotations.md).
For a general description of how to investigate failed operations, see [Improving Coverage](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/ImprovingCoverage.md).
48
-
When testing a very large API, you may find it helpful to test one failed endpoint at a time and get it to succeed, which may be specified through the [Engine Settings](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/EngineSettings.md).
61
+
For a general description of how to investigate failed operations, see [Improving Coverage](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/ImprovingCoverage.md).
62
+
When testing a very large API, you may find it helpful to test one failed endpoint at a time and get it to succeed, which may be specified through the [Engine Settings](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/EngineSettings.md).
49
63
64
+
**C. Test your changes by running RESTler locally.**
50
65
51
-
### C. Test your changes by running RESTler locally.
66
+
Compile the specifications using the command line below. ```config.json``` will refer to your updated Swagger files and examples.
52
67
53
-
Compile the specifications using the command line below. ```config.json``` will refer to your updated Swagger files and examples.
68
+
>Restler.exe compile config.json
54
69
55
-
>Restler.exe compile config.json
70
+
Test the API using the exact command line taken from the _TrafficValidation_ step log in the CI pipeline (replace the ```token_refresh_command``` below with the authentication script on your local path). For example:
56
71
57
-
Test the API using the exact command line taken from the _TrafficValidation_ step log in the CI pipeline (replace the ```token_refresh_command``` below with the authentication script on your local path). For example:
#### 3. Upload the modified restler configuration file so the PR pipeline can use it and re-run the Service Contract Accuracy report
60
75
76
+
After you have debug restler locally and improved API coverage, you can upload the modified configuration files to the azure-rest-api-specs repo so the pipeline can use them. For a specific resource provider, the restler config files is located at `restler` under the api-version folder. In swagger pipeline, the traffic validation check will load the customized restler config from the api-version folder and then run the restler test.
61
77
62
-
3. Upload the modified conf file so the PR pipeline can use it and re-run the Service Contract Accuracy report
63
-
- ...
78
+

64
79
80
+
When you uploaded the restler config files, the pipeline will re-run and re-generate the Service Contract Accuracy Report.
0 commit comments