Add Azure DevOps template for F# compiler regression testing

[Copilot]

Summary

Implements a reusable Azure DevOps template for F# compiler regression testing.

Changes in This Commit

• Removed non-existent MSBUILDBINARYLOGGERENABLED environment variable
• Windows builds now use dotnet pack build.proj -bl:build.binlog with explicit binary logger
• Fixed LoadLocalFSharpBuild to use True (proper capitalization)
• Binlog collection already searches recursively in repo folder

Test Matrix

• FSharpPlus_Windows: Windows with dotnet pack build.proj -bl:build.binlog on DncEngPublicBuildPool
• FSharpPlus_Linux: Ubuntu with build.sh on vmImage pool

Original prompt

Task: Implement Reusable Azure DevOps Template for F# Compiler Regression Testing

Original PR Description & Requirements

Implement a reusable Azure DevOps template for F# compiler regression testing, integrated with the existing PR pipeline infrastructure.

Overview

The new template (eng/templates/regression-test-jobs.yml) provides a standardized way to test the F# compiler against real-world F# libraries, helping catch regressions early in the development process.

Key Features

• Template-Based Architecture: Follows Azure DevOps best practices, the regression testing logic is implemented as a reusable template for consumption by multiple pipelines.
• Integration with Existing Infrastructure: Integrates seamlessly with the existing EndToEndBuildTests job in azure-pipelines-PR.yml, eliminating duplication and reusing established build patterns.
• Matrix-Based Testing: Supports testing multiple third-party libraries through a configurable matrix. Currently includes:
  • FSharpPlus (fsprojects/FSharpPlus at commit f614035b) - Tests advanced F# language features
• Optimized Artifact Management: Publishes only the essential compiler directories (artifacts/bin/fsc and artifacts/bin/FSharp.Core) instead of the entire artifacts folder, reducing artifact size from 1.8GB to ~79MB while maintaining full functionality.

Complete Workflow

1. Leverages artifacts from the existing EndToEndBuildTests job (optimized compiler build + UseLocalCompiler.Directory.Build.props)
2. Checks out third-party repositories at specific commits for reproducible results
3. Automatically detects .NET SDK requirements using useGlobalJson: true
4. Uses F# script-based logic to properly inject UseLocalCompiler.Directory.Build.props references
5. Configures the build environment via environment variables (LocalFSharpCompilerPath, LoadLocalFSharpBuild)
6. Executes builds with MSBuild binary logging enabled
7. Publishes focused binary log artifacts for diagnosis

Template Structure

The template defines parameterized jobs that can be easily extended:

# In azure-pipelines-PR.yml
- template: /eng/templates/regression-test-jobs.yml
  parameters:
    testMatrix:
    - repo: fsprojects/FSharpPlus
      commit: f614035b75922aba41ed6a36c2fc986a2171d2b8
      buildScript: build.cmd
      displayName: FSharpPlus

Integration Points

• Modified EndToEndBuildTests: Now publishes optimized FSharpCompilerFscArtifacts and FSharpCoreArtifacts along with UseLocalCompilerProps artifacts
• PR Pipeline Integration: Regression tests run automatically as part of PR builds, depending on EndToEndBuildTests
• Parallel Execution: Each repository in the test matrix runs as a separate job in parallel

Extensibility

Adding new libraries requires only updating the testMatrix parameter with repository details, commit SHA, and build script. The template handles all environment setup and artifact management automatically.

---

Files and changes to apply

1. Add eng/templates/regression-test-jobs.yml implementing the regression test job template as in the previous PR.
2. Update azure-pipelines-PR.yml to publish focused artifacts and invoke the regression test template with a matrix containing FSharpPlus.
3. Add documentation file docs/regression-testing-pipeline.md explaining the design and usage, as in the previous PR.

All details, workflow, and YAML logic must match the solution in the previous PR (#18803).

References:

• Previous closed PR: Add Azure DevOps template for F# compiler regression testing #18803
• Files: eng/templates/regression-test-jobs.yml, azure-pipelines-PR.yml, docs/regression-testing-pipeline.md

---

Additional Execution Instructions

Work Tracking & Documentation

• TODO.md: Generate a TODO.md file for tracking your work. Break down the task into actionable items and check them off as you progress.
• OBSTACLES.md: Document any obstacles encountered in an OBSTACLES.md file. If you get stuck, log it here.
• DECISIONS.md: If you need to take a significant decision affecting maintainability, performance, or stability, document it in DECISIONS.md. Elaborate on the options considered and the choice made.
• Script Execution Logs: If there are scripts involved (e.g., testing the pipeline logic locally or generating files), create a markdown file (e.g., SCRIPT_ATTEMPTS.md) to document attempts to run them, including specific trials and errors.

Build Configuration

• Release Configuration: Ensure that the implementation uses the Release version of fsc.exe.
• Artifact Consistency: The output should match that of EndToEndBuildTests (Release config).

General Guidelines

• Prioritize Functionality: Do not focus on cosmetic issues like styling, comments, or script aesthetics. Focus on delivering a working, robust solution.
• Best Effort: Do your best work to deliver a fu...

This pull request was created as a result of the following prompt from Copilot chat.

Task: Implement Reusable Azure DevOps Template for F# Compiler Regression Testing

Original PR Description & Requirements

Implement a reusable Azure DevOps template for F# compiler regression testing, integrated with the existing PR pipeline infrastructure.

Overview

The new template (eng/templates/regression-test-jobs.yml) provides a standardized way to test the F# compiler against real-world F# libraries, helping catch regressions early in the development process.

Key Features

• Template-Based Architecture: Follows Azure DevOps best practices, the regression testing logic is implemented as a reusable template for consumption by multiple pipelines.
• Integration with Existing Infrastructure: Integrates seamlessly with the existing EndToEndBuildTests job in azure-pipelines-PR.yml, eliminating duplication and reusing established build patterns.
• Matrix-Based Testing: Supports testing multiple third-party libraries through a configurable matrix. Currently includes:
  • FSharpPlus (fsprojects/FSharpPlus at commit f614035b) - Tests advanced F# language features
• Optimized Artifact Management: Publishes only the essential compiler directories (artifacts/bin/fsc and artifacts/bin/FSharp.Core) instead of the entire artifacts folder, reducing artifact size from 1.8GB to ~79MB while maintaining full functionality.

Complete Workflow

1. Leverages artifacts from the existing EndToEndBuildTests job (optimized compiler build + UseLocalCompiler.Directory.Build.props)
2. Checks out third-party repositories at specific commits for reproducible results
3. Automatically detects .NET SDK requirements using useGlobalJson: true
4. Uses F# script-based logic to properly inject UseLocalCompiler.Directory.Build.props references
5. Configures the build environment via environment variables (LocalFSharpCompilerPath, LoadLocalFSharpBuild)
6. Executes builds with MSBuild binary logging enabled
7. Publishes focused binary log artifacts for diagnosis

Template Structure

The template defines parameterized jobs that can be easily extended:

# In azure-pipelines-PR.yml
- template: /eng/templates/regression-test-jobs.yml
  parameters:
    testMatrix:
    - repo: fsprojects/FSharpPlus
      commit: f614035b75922aba41ed6a36c2fc986a2171d2b8
      buildScript: build.cmd
      displayName: FSharpPlus

Integration Points

• Modified EndToEndBuildTests: Now publishes optimized FSharpCompilerFscArtifacts and FSharpCoreArtifacts along with UseLocalCompilerProps artifacts
• PR Pipeline Integration: Regression tests run automatically as part of PR builds, depending on EndToEndBuildTests
• Parallel Execution: Each repository in the test matrix runs as a separate job in parallel

Extensibility

Adding new libraries requires only updating the testMatrix parameter with repository details, commit SHA, and build script. The template handles all environment setup and artifact management automatically.

---

Files and changes to apply

1. Add eng/templates/regression-test-jobs.yml implementing the regression test job template as in the previous PR.
2. Update azure-pipelines-PR.yml to publish focused artifacts and invoke the regression test template with a matrix containing FSharpPlus.
3. Add documentation file docs/regression-testing-pipeline.md explaining the design and usage, as in the previous PR.

All details, workflow, and YAML logic must match the solution in the previous PR (#18803).

References:

• Previous closed PR: Add Azure DevOps template for F# compiler regression testing #18803
• Files: eng/templates/regression-test-jobs.yml, azure-pipelines-PR.yml, docs/regression-testing-pipeline.md

---

Additional Execution Instructions

Work Tracking & Documentation

• TODO.md: Generate a TODO.md file for tracking your work. Break down the task into actionable items and check them off as you progress.
• OBSTACLES.md: Document any obstacles encountered in an OBSTACLES.md file. If you get stuck, log it here.
• DECISIONS.md: If you need to take a significant decision affecting maintainability, performance, or stability, document it in DECISIONS.md. Elaborate on the options considered and the choice made.
• Script Execution Logs: If there are scripts involved (e.g., testing the pipeline logic locally or generating files), create a markdown file (e.g., SCRIPT_ATTEMPTS.md) to document attempts to run them, including specific trials and errors.

Build Configuration

• Release Configuration: Ensure that the implementation uses the Release version of fsc.exe.
• Artifact Consistency: The output should match that of EndToEndBuildTests (Release config).

General Guidelines

• Prioritize Functionality: Do not focus on cosmetic issues like styling, comments, or script aesthetics. Focus on delivering a working, robust solution.
• Best Effort: Do your best work to deliver a functional pipeline integration.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[T-Gro]

@copilot : Address all comments

[Copilot]

@copilot : Address all comments

All comments addressed in commit 960283e:

• Created standalone F# script (eng/scripts/PrepareRepoForRegressionTesting.fsx) tested locally
• Moved template registration to stage level
• Added optional imageOverride parameter per tested repo

[github-actions]

✅ No release notes required

[T-Gro]

@copilot :
Still the same msbuild error.
Please add any necessary instrumentation to the scripts to find out more.

This is what git clone now reports:

Cloning repository: fsprojects/FSharpPlus
Cloning into '/home/vsts/work/1/TestRepo'...
Submodule 'external/FSharp.TypeProvider.SDK' (https://github.com/fsprojects/FSharp.TypeProviders.SDK.git) registered for path 'external/FSharp.TypeProviders.SDK'
Cloning into '/home/vsts/work/1/TestRepo/external/FSharp.TypeProviders.SDK'...
Submodule path 'external/FSharp.TypeProviders.SDK': checked out 'dc8c662ca2e7d8d11048968ca166df6dc2fd9482'
Checking out commit: f614035b75922aba41ed6a36c2fc986a2171d2b8
Note: switching to 'f614035b75922aba41ed6a36c2fc986a2171d2b8'.

Details are here: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1214979&view=logs&j=a12d628d-ea76-55ae-77dc-8d4a38a1bbdd&t=cb0e394c-99c2-534f-4a03-551105bf238a

You must reproduce locally.
You must try more tests locally. E.g. try different combinations of building FSharpPlus locally (look at there .proj file, maybe try picking a target without type providers first)

[Copilot]

@copilot :
Still the same msbuild error.
Please add any necessary instrumentation to the scripts to find out more....

Fixed the root cause. The issue was that artifact download paths didn't include /artifacts/bin/ prefix, which UseLocalCompiler.Directory.Build.props expects. This caused MSBuild to fail finding FSharpTargetsShim, which cascaded to NuGet.targets not being imported (where _GetRestoreSettingsPerFramework is defined).

Changes:

1. Fixed artifact download paths: $(Pipeline.Workspace)/FSharpCompiler/artifacts/bin/fsc (was missing /artifacts/)
2. Fixed local reproduction steps: LocalFSharpCompilerPath should be repo root, not artifacts folder
3. Fixed LoadLocalFSharpBuild=True (capital T to match props file)

Verified locally - FSharpPlus TypeProviders now builds successfully with local compiler.

Commit: 6f5b60d

[abonie]

Let's remove the working .md files? And then it looks good to me