Skip to content

Commit 08560e0

Browse files
authored
Added traffic validation document (Azure#20271)
* Added traffic validation document * Update per the feedback
1 parent 7a2aa12 commit 08560e0

File tree

2 files changed

+28
-0
lines changed

2 files changed

+28
-0
lines changed

documentation/ci-fix.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -165,6 +165,13 @@ This validator is to ensure the cadl & swagger files in PR are consistent and th
165165
|InConsistentSwagger| the generated swagger is inconsistent with the swagger in PR, so you need to re-generate swagger from cadl, and check in it |
166166
|SwaggerNotExistInPR| the occurs when there is cadl file in the PR but the swagger is not present in the PR, so you need to add the swagger to the PR |
167167

168+
## Traffic Validation
169+
170+
It generates traffics for all operations defined in swaggers under default tag of readme.md by using [RESTLer](https://github.com/microsoft/restler-fuzzer). Then it validates the request and response pairs from these traffics against correspondent swaggers. Finally, it provides a html report that reflects swagger accuracy.
171+
172+
### How to understand and improve the report
173+
Please refer to [swagger-accuracy-report](./swagger-accuracy-report.md).
174+
168175
## Suppression Process
169176

170177
In case there are validation errors reported against your service that you believe do not apply, we have a suppression process you can follow to permanently remove these reported errors for your specs. Refer to [Swagger Suppression Process](https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/85/Swagger-Suppression-Process) for detailed guidance.
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Swagger Accuracy Report
2+
## What is this report about
3+
[@Jeffrey to add details]
4+
5+
## How to improve report
6+
There're two parts of the report need to be improved if it has content.
7+
* First part is `Failed Operations` which includes the schema validation errors for listed operations, all the errors are supposed to be fixed. From the error code you could browse to the linked document about the error defintion and fix tips.
8+
* Second part is `Untested Operations` which includes the operations which don't have the traffic generated by RESTler for validation. It could be because the dependent operations fails in validation in which case please fix failed operation firstly so that this untested operation will be unlocked automatically.
9+
10+
## How to improve Untested Operations
11+
All the traffics for each operation are generated by RESTler automatically, if the result says the operation is untested then you could try to modify RESTler config files to improve RESTler generation coverage.
12+
[@Marina to add details on how to modify configs]
13+
14+
* Download and modify each RESTler config
15+
[to add details]
16+
17+
* Debug locally with updated RESTler configs
18+
[to add details]
19+
20+
* Check in updated RESTler configs to be used in swagger PR pipeline and re-run
21+
[to add details]

0 commit comments

Comments
 (0)