Skip to content

Commit 2beb247

Browse files
MoChiliaisra-fel
andauthored
[StaticAnalysis] CI ExampleAnalysis (Azure#18454)
* add scriptAnalyzer * test for change cs * test for change both .cs and .md * fix a bug: error when no changed file * fix output of result * test for wrong md * fix import-module * sync up with shiying/ci-example * output by issueChecker * fix the bug for exceptionFilePath * Report errors according to the .csv * Test output of scriptAnalyzer * test * restore * fix bug for module name * fix bug for module name * build dependent-module for md; complete csv; output in one ps1 * illustration for example issue; add suppress; change problemId * Update Program.cs * Update Measure-MarkdownOrScript.ps1 * add suppress function * Delete ExampleAnalyzer.cs * Update ParameterNameAndValue.psm1 fix a bug for not found module name * skip autogenerated example * fix output; output missing errors for placeholders; fix description * fix bugs * Combine install platyPS and PSScriptAnalyzer together into a "Install PowerShell Dependencies" * abandon Jenkins for CI * Apply suggestions from code review Co-authored-by: Yeming Liu <11371776+isra-fel@users.noreply.github.com> * comment for problemID * Update analyze-steps.yml * use absolute path instead of relative path * output unexcepted error * fix a bug * optimize writing * Update tools/StaticAnalysis/ExampleAnalyzer/utils.ps1 Co-authored-by: Yeming Liu <11371776+isra-fel@users.noreply.github.com> * change regular expression for exampletitle * restore * change severity * change severity * change comment * match autogenerated examples * fix getting function name Co-authored-by: Yeming Liu <11371776+isra-fel@users.noreply.github.com>
1 parent 66b6c26 commit 2beb247

File tree

18 files changed

+1508
-17
lines changed

18 files changed

+1508
-17
lines changed

.azure-pipelines/util/analyze-steps.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,9 @@ steps:
2929
packageType: sdk
3030
version: 3.1.x
3131

32-
- pwsh: 'Install-Module platyPS -Force -Confirm:$false -Scope CurrentUser'
33-
displayName: 'Install platyPS'
34-
32+
- pwsh: 'Install-Module "platyPS", "PSScriptAnalyzer" -Force -Confirm:$false -Scope CurrentUser'
33+
displayName: 'Install PowerShell Dependencies'
34+
3535
- task: DotNetCoreCLI@2
3636
displayName: 'Generate Help'
3737
inputs:

.ci-config.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@
8787
"src/{ModuleName}/**/*.md$"
8888
],
8989
"phases": [
90-
"build:module",
90+
"build:dependent-module",
9191
"help:module"
9292
]
9393
},

build.proj

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,7 @@
118118
<Exec Command="dotnet publish $(RepoTools)BuildPackagesTask/Microsoft.Azure.Build.Tasks/Microsoft.Azure.Build.Tasks.csproj -c $(Configuration)" />
119119

120120
<!-- Get all of the files changed in the given pull request -->
121-
<FilesChangedTask RepositoryOwner="Azure" RepositoryName="azure-powershell" PullRequestNumber="$(PullRequestNumber)" TargetModule="$(TargetModule)">
121+
<FilesChangedTask RepositoryOwner="Azure" RepositoryName="azure-powershell" PullRequestNumber="$(PullRequestNumber)" TargetModule="$(TargetModule)" OutputFile="$(RepoArtifacts)/FilesChanged.txt">
122122
<Output TaskParameter="FilesChanged" ItemName="FilesChanged" />
123123
</FilesChangedTask>
124124

@@ -263,7 +263,12 @@
263263
<Exec Command="dotnet $(RepoArtifacts)StaticAnalysis/StaticAnalysis.Netcore.dll -p $(RepoArtifacts)$(Configuration) -r $(StaticAnalysisOutputDirectory) --analyzers help -u -m '%(FilterTaskResult.help)'" />
264264
</Target>
265265

266-
<Target Name="StaticAnalysis" DependsOnTargets="StaticAnalysisBreakingChange;StaticAnalysisDependency;StaticAnalysisSignature;StaticAnalysisHelp">
266+
<Target Name="StaticAnalysisExample" Condition="'$(RunStaticAnalysis)' == 'true'" DependsOnTargets="Build" AfterTargets="StaticAnalysisHelp">
267+
<Message Importance="high" Text="Running static analysis for PowerShell examples..." />
268+
<Exec Command="$(PowerShellCoreCommandPrefix) &quot;. $(RepoTools)/StaticAnalysis/ExampleAnalyzer/Measure-MarkdownOrScript.ps1 -MarkdownPaths $(RepoArtifacts)/FilesChanged.txt -RulePaths $(RepoTools)/StaticAnalysis/ExampleAnalyzer/AnalyzeRules/*.psm1 -Recurse -AnalyzeScriptsInFile -OutputScriptsInFile -OutputResultsByModule &quot;"/>
269+
</Target>
270+
271+
<Target Name="StaticAnalysis" DependsOnTargets="StaticAnalysisBreakingChange;StaticAnalysisDependency;StaticAnalysisSignature;StaticAnalysisHelp;StaticAnalysisExample">
267272
<Message Importance="high" Text="Running static analysis..." />
268273

269274
<Exec Command="$(PowerShellCoreCommandPrefix) &quot;. $(RepoTools)/CheckAssemblies.ps1 -BuildConfig $(Configuration) &quot;" />

documentation/Debugging-StaticAnalysis-Errors.md

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Our StaticAnalysis tools help us ensure our modules follow PowerShell guidelines
88
- [Breaking Changes](#breaking-changes)
99
- [Signature Issues](#signature-issues)
1010
- [Help Issues](#help-issues)
11+
- [Example Issues](#example-issues)
1112

1213
## How to know if you have a StaticAnalysis Error
1314
If your build is failing, click on the Jenkins job inside the PR (marked as "Default" within checks). Then check the Console Output within the Jenkins job. If you have this error, then you have failed StaticAnalysis:
@@ -34,21 +35,32 @@ If you make a change that could cause a breaking change, it will be listed in `B
3435

3536
_Note_: Sometimes the error listed in the .csv file can be a false positive (for example, if you change a parameter attribute to span all parameter sets rather than individual parameter sets). Please read the error thoroughly and examine the relevant code before deciding that an error is a false positive, and contact the Azure PowerShell team if you have questions. If you are releasing a preview module, are releasing during a breaking change release, or have determined that the error is a false positive, please follow these instructions to suppress the errors:
3637

37-
- Download the `BreakingChangeIssues.csv` file from the Jenkins build
38+
- Download the `BreakingChangeIssues.csv` file from the CI pipeline artifacts
3839
- Open the file using a text editor (such as VS Code) and copy each of the errors you'd like to suppress
3940
- Paste each of these errors into the `BreakingChangeIssues.csv` file found in their respective [module folder](../tools/StaticAnalysis/Exceptions) (_e.g._, if a breaking change is being suppressed for Compute, then you would paste the corresponding line(s) in the `tools/StaticAnalysis/Exceptions/Az.Compute/BreakingChangeIssues.csv` file) using the same text editor
40-
- Push the changes to the .csv file and ensure the errors no longer show up in the `BreakingChangeIssues.csv` file output from the Jenkins build.
41+
- Push the changes to the .csv file and ensure the errors no longer show up in the `BreakingChangeIssues.csv` file output from the CI pipeline artifacts.
4142

4243
We take breaking changes very seriously, so please be mindful about the violations that you suppress in our repo.
4344

4445
### Signature Issues
4546
Signature issues occur when your cmdlets do not follow PowerShell standards. Please check the [_Cmdlet Best Practices_](https://github.com/Azure/azure-powershell/blob/main/documentation/development-docs/design-guidelines/cmdlet-best-practices.md) and the [_Parameter Best Practices_](https://github.com/Azure/azure-powershell/blob/main/documentation/development-docs/design-guidelines/parameter-best-practices.md) documents to ensure you are following PowerShell guidelines. Issues with severity 0 or 1 must be addressed, while issues with severity 2 are advisory. If you have an issue with severity 0 or 1 that has been approved by the Azure PowerShell team, you can suppress them following these steps:
4647

47-
- Download the `SignatureIssues.csv` file from the Jenkins build
48+
- Download the `SignatureIssues.csv` file from the CI pipeline artifacts
4849
- Open the file using a text editor (such as VS Code) and copy each of the errors you'd like to suppress
4950
- Paste each of these errors into the `SignatureIssues.csv` file found in their respective [module folder](../tools/StaticAnalysis/Exceptions) (_e.g.,_ if a signature issue is being suppressed for Sql, then you would paste the corresponding line(s) in the `tools/StaticAnalysis/Exceptions/Az.Sql/SignatureIssues.csv` file) using the same text editor
50-
- Copy each of the errors you would like to suppress directly from the SignatureIssues.csv file output in the Jenkins build
51-
- Push the changes to the .csv file and ensure the errors no longer show up in the `SignatureIssues.csv` file output from the Jenkins build.
51+
- Copy each of the errors you would like to suppress directly from the SignatureIssues.csv file output in the CI pipeline artifacts
52+
- Push the changes to the .csv file and ensure the errors no longer show up in the `SignatureIssues.csv` file output from the CI pipeline artifacts.
5253

5354
### Help Issues
5455
Most help issues that cause StaticAnalysis to fail occur when help has not been added for a particular cmdlet. If you have not generated help for your new cmdlets, please follow the instructions [here](https://github.com/Azure/azure-powershell/blob/main/documentation/development-docs/help-generation.md). If this is not the issue, follow the steps listed under "Remediation" for each violation listed in HelpIssues.csv.
56+
57+
### Example Issues
58+
Example issues occur when your changed markdown files in the `help` folder (_e.g.,_ `src/Accounts/Accounts/help`) violate PowerShell language best practices. Please follow the suggestion displayed in "Remediation" entry for each violation listed in `ExampleIssues.csv`. If you have an issue with severity 0 or 1 that has been approved by the Azure PowerShell team, you can suppress them following these steps:
59+
60+
- Download the `ExampleIssues.csv` file from the CI pipeline artifacts
61+
- Open the file using a text editor (such as VS Code) and copy each of the errors you'd like to suppress
62+
- Paste each of these errors into the `ExampleIssues.csv` file found in their respective [module folder](../tools/StaticAnalysis/Exceptions) (_e.g.,_ if an example issue is being suppressed for Accounts, then you would paste the corresponding line(s) in the `tools/StaticAnalysis/Exceptions/Az.Accounts/ExampleIssue.csv` file) using the same text editor
63+
- Copy each of the errors you would like to suppress directly from the ExampleIssues.csv file output in the CI pipeline artifacts
64+
- Push the changes to the .csv file and ensure the errors no longer show up in the `ExampleIssues.csv` file output from the CI pipeline artifacts.
65+
66+
To better standardize the writing of documents, please also check the warning issues with severity 2 by downloading the `ExampleIssues.csv` file.

documentation/tooling/static-analysis.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,8 @@ The dependency analyzer can be found in the [`DependencyAnalyzer`](https://githu
8585
- The implementation of the `IReportRecord` interface; defines what a missing assembly exception looks like when it's reported in the `MissingAssembly.csv` file that is found in the build artifacts of a CI run, as well as how to compare a new record to a record found in the existing `MissingAssembly.csv` file used for exception suppressions
8686
- `SharedAssemblyConflict`
8787
- The implementation of the `IReportRecord` interface; defines what a shared conflict exception looks like when it's reported in the `SharedAssemblyConflict.csv` file that is found in the build artifacts of a CI run, as well as how to compare a new record to a record found in the existing `SharedAssemblyConflict.csv` file used for exception suppressions
88+
- `ExampleIssue`
89+
- The implementation of the `IReportRecord` interface; defines what an example issue exception looks like when it's reported in the `ExampleIssues.csv` file that is found in the build artifacts of a CI run, as well as how to compare a new record to a record found in the existing `ExampleIssues.csv` file used for exception suppressions
8890

8991
#### Help Analyzer
9092

tools/BuildPackagesTask/Microsoft.Azure.Build.Tasks/FilesChangedTask.cs

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,11 @@ public class FilesChangedTask : Task
4949
/// </summary>
5050
public string TargetModule { get; set; }
5151

52+
/// <summary>
53+
/// Gets or set the OutputFile, store FilesChanged.txt in 'artifacts' folder
54+
/// </summary>
55+
public string OutputFile { get; set; }
56+
5257
/// <summary>
5358
/// Gets or sets the files changed produced by the task.
5459
/// </summary>
@@ -148,10 +153,10 @@ public override bool Execute()
148153
return true;
149154
}
150155

151-
// This method will record the changed files into FilesChanged.txt under root folder for other task to consum.
156+
// This method will record the changed files into a text file at `OutputFile` for other task to consum.
152157
private void SerializeChangedFilesToFile(string[] FilesChanged)
153158
{
154-
File.WriteAllLines("FilesChanged.txt", FilesChanged);
159+
File.WriteAllLines(OutputFile, FilesChanged);
155160
}
156161
}
157162
}

tools/PrepareAutorestModule.ps1

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717

1818
param(
1919
)
20-
$ChangedFiles = Get-Content -Path "$PSScriptRoot\..\FilesChanged.txt"
20+
$ChangedFiles = Get-Content -Path "$PSScriptRoot\..\artifacts\FilesChanged.txt"
2121

2222
$ALL_MODULE = "ALL_MODULE"
2323

tools/PrepareForSecurityCheck.ps1

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717

1818
param(
1919
)
20-
$ChangedFiles = Get-Content -Path "$PSScriptRoot\..\FilesChanged.txt"
20+
$ChangedFiles = Get-Content -Path "$PSScriptRoot\..\artifacts\FilesChanged.txt"
2121

2222
$SecurityTmpFolder = "$PSScriptRoot\..\SecurityTmp"
2323
New-Item -ItemType Directory -Force -Path $SecurityTmpFolder
Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
<#
2+
.SYNOPSIS
3+
Custom rule for command name.
4+
.NOTES
5+
File: CommandName.psm1
6+
#>
7+
8+
enum RuleNames {
9+
Invalid_Cmdlet
10+
Is_Alias
11+
Capitalization_Conventions_Violated
12+
}
13+
14+
<#
15+
.SYNOPSIS
16+
Returns invaild, alias or unrecognized cmdlets.
17+
#>
18+
function Measure-CommandName {
19+
[CmdletBinding()]
20+
[OutputType([Microsoft.Windows.PowerShell.ScriptAnalyzer.Generic.DiagnosticRecord[]])]
21+
param(
22+
[Parameter(Mandatory)]
23+
[ValidateNotNullOrEmpty()]
24+
[System.Management.Automation.Language.ScriptBlockAst]
25+
$ScriptBlockAst
26+
)
27+
begin{
28+
$modulePath = "$PSScriptRoot\..\..\..\..\artifacts\Debug\Az.*\Az.*.psd1"
29+
Get-Item $modulePath | Import-Module -Global
30+
}
31+
process {
32+
$Results = @()
33+
$global:CommandParameterPair = @()
34+
$global:Ast = $null
35+
36+
try {
37+
[ScriptBlock]$Predicate = {
38+
param([System.Management.Automation.Language.Ast]$Ast)
39+
$global:Ast = $Ast
40+
41+
#Find all command in .ps1
42+
if ($Ast -is [System.Management.Automation.Language.CommandAst]) {
43+
[System.Management.Automation.Language.CommandAst]$CommandAst = $Ast
44+
# Get wrapper function name by command element
45+
$funcAst = $CommandAst
46+
while($funcAst -isnot [System.Management.Automation.Language.FunctionDefinitionAst] -and $null -ne $funcAst.Parent.Parent.Parent){
47+
$funcAst = $funcAst.Parent
48+
}
49+
$ModuleCmdletExNum = $funcAst.name
50+
51+
if ($CommandAst.InvocationOperator -eq "Unknown") {
52+
$CommandName = $CommandAst.CommandElements[0].Extent.Text
53+
$GetCommand = Get-Command $CommandName -ErrorAction SilentlyContinue
54+
if ($null -eq $GetCommand) {
55+
# CommandName is not valid.
56+
$global:CommandParameterPair += @{
57+
CommandName = $CommandName
58+
ParameterName = "<is not valid>"
59+
ModuleCmdletExNum = $ModuleCmdletExNum
60+
}
61+
return $true
62+
}
63+
else {
64+
if ($GetCommand.CommandType -eq "Alias") {
65+
# CommandName is an alias.
66+
$global:CommandParameterPair += @{
67+
CommandName = $CommandName
68+
ParameterName = "<is an alias>"
69+
ModuleCmdletExNum = $ModuleCmdletExNum
70+
}
71+
return $true
72+
}
73+
if ($CommandName -cnotmatch "^([A-Z][a-z]+)+-([A-Z][a-z0-9]*)+$") {
74+
# CommandName doesn't follow the Capitalization Conventions.
75+
$global:CommandParameterPair += @{
76+
CommandName = $CommandName
77+
ParameterName = "<doesn't follow the Capitalization Conventions>"
78+
ModuleCmdletExNum = $ModuleCmdletExNum
79+
}
80+
return $true
81+
}
82+
}
83+
}
84+
}
85+
86+
return $false
87+
}
88+
89+
# Find all false scriptblock
90+
[System.Management.Automation.Language.Ast[]]$Asts = $ScriptBlockAst.FindAll($Predicate, $false)
91+
for ($i = 0; $i -lt $Asts.Count; $i++) {
92+
if ($global:CommandParameterPair[$i].ParameterName -eq "<is not valid>") {
93+
$Message = "$($CommandParameterPair[$i].CommandName) is not a valid command name."
94+
$RuleName = [RuleNames]::Invalid_Cmdlet
95+
$RuleSuppressionID = "5000"
96+
$Remediation = "Check the spell of $($CommandParameterPair[$i].CommandName)."
97+
$Severity = "Error"
98+
}
99+
if ($global:CommandParameterPair[$i].ParameterName -eq "<is an alias>") {
100+
$Message = "$($CommandParameterPair[$i].CommandName) is an alias of `"$((Get-Alias $CommandParameterPair[$i].CommandName)[0].ResolvedCommandName)`"."
101+
$RuleName = [RuleNames]::Is_Alias
102+
$RuleSuppressionID = "5100"
103+
$Remediation = "Use formal name `"$((Get-Alias $CommandParameterPair[$i].CommandName)[0].ResolvedCommandName)`" of the alias `"$($CommandParameterPair[$i].CommandName)`"."
104+
$Severity = "Warning"
105+
}
106+
if ($global:CommandParameterPair[$i].ParameterName -eq "<doesn't follow the Capitalization Conventions>") {
107+
$Message = "$($CommandParameterPair[$i].CommandName) doesn't follow the Capitalization Conventions."
108+
$RuleName = [RuleNames]::Capitalization_Conventions_Violated
109+
$RuleSuppressionID = "5101"
110+
$name = $($CommandParameterPair[$i].CommandName)
111+
$textInfo = (Get-Culture).TextInfo
112+
$CorrectName = $textInfo.ToTitleCase(($name -split "-")[0])
113+
$CorrectName += "-Az"
114+
$CorrectName += $textInfo.ToTitleCase(($name -split "Az")[1])
115+
$Remediation = "Check the Capitalization Conventions. Suggest format: $CorrectName"
116+
$Severity = "Warning"
117+
}
118+
$ModuleCmdletExNum = $($CommandParameterPair[$i].ModuleCmdletExNum)
119+
$Result = [Microsoft.Windows.PowerShell.ScriptAnalyzer.Generic.DiagnosticRecord]@{
120+
Message = "$ModuleCmdletExNum-@$Message@$Remediation";
121+
Extent = $Asts[$i].Extent;
122+
RuleName = $RuleName;
123+
Severity = $Severity
124+
RuleSuppressionID = $RuleSuppressionID
125+
}
126+
$Results += $Result
127+
}
128+
return $Results
129+
}
130+
catch {
131+
$Result = [Microsoft.Windows.PowerShell.ScriptAnalyzer.Generic.DiagnosticRecord]@{
132+
Message = $_.Exception.Message;
133+
Extent = $global:Ast.Extent;
134+
RuleName = $PSCmdlet.MyInvocation.InvocationName;
135+
Severity = "Error"
136+
}
137+
$Results += $Result
138+
return $Results
139+
}
140+
}
141+
}
142+
143+
Export-ModuleMember -Function Measure-*

0 commit comments

Comments
 (0)