Skip to content

Commit 151d278

Browse files
ruowanmikekistler
andauthored
update download restler config part (#20408)
* update download restler config part * Added upload modified restler config and fix markdown format * Update documentation/swagger-accuracy-report.md Co-authored-by: Mike Kistler <mikekistler@microsoft.com> Co-authored-by: Mike Kistler <mikekistler@microsoft.com>
1 parent b51a5fa commit 151d278

File tree

4 files changed

+46
-30
lines changed

4 files changed

+46
-30
lines changed
115 KB
Loading
53 KB
Loading
32.3 KB
Loading

documentation/swagger-accuracy-report.md

Lines changed: 46 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -4,61 +4,77 @@ The purpose of the Service Contract (Swagger) Accuracy Report is to ensure that
44

55
Our tooling ([RESTler](https://github.com/microsoft/restler-fuzzer) and [OAV](https://github.com/azure/oav)) that produces this report tests all service operations and compares the actual requests/responses against the contract for accuracy.
66

7-
***
7+
---
8+
89
**IMPORTANT: _All operations must be tested and pass before the PR contract can be merged._**
9-
***
10+
11+
---
1012

1113
For any `Failed Operations`, use the error information (and links provided) to either fix the contract or fix the service behavior. Fix failed operations first as this may enable our tooling to test some of the currently untested operations.
1214

1315
For any `Untested Operations`, you must modify the RESTler configuration files to improve API coverage.
1416

15-
1. Download the RESTler configuration files
16-
- ...
17+
## Steps to improve API coverage
18+
19+
### 1. Download the RESTler configuration files
20+
21+
In swagger PR pipeline, traffic validation pipeline will generate RESTler configuration files as artifacts. You could click the **pipeline artifact** to download the configuration files.
22+
23+
![traffic-validation-pipeline-comment](./images/traffic-validation-pipeline-comment.png)
24+
25+
Here is an example of the artifacts generated by the traffic validation pipeline:
26+
27+
![traffic-validation-pipeline-artifacts](./images/traffic-validation-pipeline-artifacts.png)
28+
29+
The red block in the above image is the generated restler configuration files and swaggerAccuracyReport. After downloading the configuration files, you could modify the configuration files and run restler locally to improve API coverage.
30+
1731
<!--
1832
TODO: the RESTler config files do not include the authentication script. How to set up authentication / where to get the script needs to be documented for a local setup.
1933
-->
2034

21-
2. Modify the configuration files to successfully execute more operations
35+
### 2. Modify the configuration files to successfully execute more operations
2236

23-
In this CI pipeline, RESTler is running in [Test](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/Testing.md) mode,
24-
where it tries to execute each operation in the Swagger specification successfully once.
25-
When an operation has dependencies (e.g. parent resources),
26-
RESTler tries to infer these dependencies and create the required parent resource.
27-
To learn more about how RESTler works, please refer to the [README](https://github.com/microsoft/restler-fuzzer).
37+
In this CI pipeline, RESTler is running in [Test](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/Testing.md) mode,
38+
where it tries to execute each operation in the Swagger specification successfully once.
39+
When an operation has dependencies (e.g. parent resources),
40+
RESTler tries to infer these dependencies and create the required parent resource.
41+
To learn more about how RESTler works, please refer to the [README](https://github.com/microsoft/restler-fuzzer).
2842

29-
You may either debug your changes locally by downloading RESTler, or for simpler fixes re-submit changes directly to the pipeline (in the latter case, refer to Step 3).
30-
The advantage of a local setup is that you will get more familiar with RESTler, and can compare your known-good local setup
31-
for testing the service with the setup in the CI pipeline to diagnose why a pipeline failure occurred.
43+
You may either debug your changes locally by downloading RESTler, or for simpler fixes re-submit changes directly to the pipeline (in the latter case, refer to Step 3).
44+
The advantage of a local setup is that you will get more familiar with RESTler, and can compare your known-good local setup
45+
for testing the service with the setup in the CI pipeline to diagnose why a pipeline failure occurred.
3246

33-
<b>To run RESTler locally:</b>
47+
<b>To run RESTler locally:</b>
3448

35-
### A. Download RESTler
49+
**A. Download RESTler**
3650

37-
See the [README](https://github.com/microsoft/restler-fuzzer) for instructions.
51+
See the [README](https://github.com/microsoft/restler-fuzzer) for instructions.
3852

39-
### B. Go through each failed operation in the RESTler logs and fix the configuration to make it succeed
53+
**B. Go through each failed operation in the RESTler logs and fix the configuration to make it succeed**
4054

41-
There are 3 configuration files that you may need to modify:
55+
There are 3 configuration files that you may need to modify:
4256

43-
- Dictionary (```dict.json```) - specify "magic constants" to assign to specific parameters. See [Dictionary](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/FuzzingDictionary.md).
44-
- Annotation file (```annotations.json```) - specify dependencies between requests. See [Annotations](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/Annotations.md).
45-
- Engine settings (```engine_settings.json```) - configure miscellaneous runtime testing parameters. See [Engine Settings](github).
57+
- Dictionary (`dict.json`) - specify "magic constants" to assign to specific parameters. See [Dictionary](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/FuzzingDictionary.md).
58+
- Annotation file (`annotations.json`) - specify dependencies between requests. See [Annotations](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/Annotations.md).
59+
- Engine settings (`engine_settings.json`) - configure miscellaneous runtime testing parameters. See [Engine Settings](github).
4660

47-
For a general description of how to investigate failed operations, see [Improving Coverage](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/ImprovingCoverage.md).
48-
When testing a very large API, you may find it helpful to test one failed endpoint at a time and get it to succeed, which may be specified through the [Engine Settings](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/EngineSettings.md).
61+
For a general description of how to investigate failed operations, see [Improving Coverage](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/ImprovingCoverage.md).
62+
When testing a very large API, you may find it helpful to test one failed endpoint at a time and get it to succeed, which may be specified through the [Engine Settings](https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/EngineSettings.md).
4963

64+
**C. Test your changes by running RESTler locally.**
5065

51-
### C. Test your changes by running RESTler locally.
66+
Compile the specifications using the command line below. ```config.json``` will refer to your updated Swagger files and examples.
5267

53-
Compile the specifications using the command line below. ```config.json``` will refer to your updated Swagger files and examples.
68+
>Restler.exe compile config.json
5469
55-
>Restler.exe compile config.json
70+
Test the API using the exact command line taken from the _TrafficValidation_ step log in the CI pipeline (replace the ```token_refresh_command``` below with the authentication script on your local path). For example:
5671

57-
Test the API using the exact command line taken from the _TrafficValidation_ step log in the CI pipeline (replace the ```token_refresh_command``` below with the authentication script on your local path). For example:
72+
>Restler.exe test --dictionary_file dict.json --grammar_file grammar.py --settings engine_settings.json --token_refresh_command "/local-scripts/get-token.sh" --token_refresh_interval 60
5873
59-
>Restler.exe test --dictionary_file dict.json --grammar_file grammar.py --settings engine_settings.json --token_refresh_command "/local-scripts/get-token.sh" --token_refresh_interval 60
74+
#### 3. Upload the modified restler configuration file so the PR pipeline can use it and re-run the Service Contract Accuracy report
6075

76+
After you have debug restler locally and improved API coverage, you can upload the modified configuration files to the azure-rest-api-specs repo so the pipeline can use them. For a specific resource provider, the restler config files is located at `restler` under the api-version folder. In swagger pipeline, the traffic validation check will load the customized restler config from the api-version folder and then run the restler test.
6177

62-
3. Upload the modified conf file so the PR pipeline can use it and re-run the Service Contract Accuracy report
63-
- ...
78+
![](./images/check-in-restler-config.png)
6479

80+
When you uploaded the restler config files, the pipeline will re-run and re-generate the Service Contract Accuracy Report.

0 commit comments

Comments
 (0)