evaluator: Introduce the option to evaluate a licenseRule for all license sources at once #11154

fviernau · 2025-11-27T20:55:23Z

See individual commits.

This is howit looks if new API is used. But migration is not necessary as its backwards compatible, it
also shows how to combine sources for one rule: oss-review-toolkit/ort-config#370.

Here you can see how the combined license soure violation is rendered in static-html reports:

plugins/commands/evaluator/src/main/kotlin/EvaluateCommand.kt

evaluator/src/main/kotlin/PackageRule.kt

evaluator/src/main/kotlin/Rule.kt

codecov · 2025-11-28T11:40:39Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.43%. Comparing base (08e631b) to head (72fa425).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #11154   +/-   ##
=========================================
  Coverage     57.43%   57.43%           
- Complexity     1701     1703    +2     
=========================================
  Files           346      346           
  Lines         12845    12845           
  Branches       1222     1222           
=========================================
  Hits           7378     7378           
  Misses         4994     4994           
  Partials        473      473

Flag	Coverage Δ
funTest-external-tools	`13.66% <ø> (ø)`
funTest-no-external-tools	`31.07% <ø> (+0.09%)`	⬆️
test-ubuntu-24.04	`42.46% <ø> (ø)`
test-windows-2025	`42.44% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sschuberth

I like the general idea of this change.

evaluator/src/main/kotlin/Rule.kt

model/src/main/kotlin/RuleViolation.kt

sschuberth · 2025-11-28T13:14:49Z

evaluator/src/main/kotlin/PackageRule.kt


        /**
-         * The source of the license.
+         * The license sources to evaluate the license for. Must not be empty and contained in the [resolvedLicense].


Nit: I'm unsure about the "to evaluate the license for" part. No one forces the rule to actually look at / evaluate the license sources. I think it should rather say "The sources where this particular license was found".

I believe "The sources where this particular license was found" is mis-leading:

The above resolvedLicense.source is actually the set of sources
where the license was found.

And licenseSources in fact are the
sources which shall be considered by the license rule.

Does this make more sense now?

evaluator/src/main/kotlin/PackageRule.kt

sschuberth · 2025-11-28T13:20:58Z

evaluator/src/main/kotlin/PackageRule.kt

+    fun licenseRule(
+        name: String,
+        licenseView: LicenseView,
+        separateEvaluationPerSource: Boolean = true,


Do we even need this configurability? Why not make the new the default, and require rule users to manually create multiple violations from this rule (one violation per source) if needed?

This is needed for:

Backwards compatibility

I believe separate violations per source to be very valuable and should be still supported in
the same simple way as before.

sschuberth · 2025-11-28T13:22:36Z

evaluator/src/main/kotlin/PackageRule.kt

            }
        }

+        /** Backwards compatibility */


Personally, I would be fine with omitting this and having a breaking change, and clearly documenting the migration path as part of the release notes.

Do you mean only this property, or the entire commit?

(Would you also be ok, with marking it as deprected now, and removing it later? This would allow a smooth transition outside of the critical path of upgrading ORT.)

Do you mean only this property, or the entire commit?

I was referring to the entire commit.

Would you also be ok, with marking it as deprected now, and removing it later?

Yes. I was just trying to say that this additional work would not be required from my side.

I ended up not deprecating this property, because the separate per source evaluation / single source per license rule or violation is still a valid (and in my opinion even the preferred) use case. In that case using that property is a bit simpler.

Agree with @fviernau we don't want to get rid of per license source evaluator per rule we just want to offer ORT user more power/flexibility to define rules that match their organization's open source policies.

Prepare for combining rule violations that stem from the same license and package, but from different licenses sources. Signed-off-by: Frank Viernau <frank.viernau@gmail.com>

Prepare for an upcoming change. Signed-off-by: Frank Viernau <frank.viernau@gmail.com>

While it is valuable to be able to evaluate the license rule seperately per license source, some users do desire to evaluate only once per license for all source. Introduce a toggle, which allows for that while keeping the old behavior as default. Signed-off-by: Frank Viernau <frank.viernau@gmail.com>

Ensure the previous change is backwards compatible. Signed-off-by: Frank Viernau <frank.viernau@gmail.com>

fviernau force-pushed the violation-multiple-license-sources branch from 734e7b1 to 7c3325f Compare November 27, 2025 20:56

github-advanced-security bot found potential problems Nov 27, 2025

View reviewed changes

plugins/commands/evaluator/src/main/kotlin/EvaluateCommand.kt Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Nov 28, 2025

View reviewed changes

evaluator/src/main/kotlin/PackageRule.kt Fixed Show fixed Hide fixed

evaluator/src/main/kotlin/Rule.kt Fixed Show fixed Hide fixed

evaluator/src/main/kotlin/Rule.kt Fixed Show fixed Hide fixed

evaluator/src/main/kotlin/Rule.kt Fixed Show fixed Hide fixed

fviernau force-pushed the violation-multiple-license-sources branch 2 times, most recently from 1e4d928 to 260566b Compare November 28, 2025 10:11

fviernau marked this pull request as ready for review November 28, 2025 10:22

fviernau requested a review from a team as a code owner November 28, 2025 10:22

fviernau force-pushed the violation-multiple-license-sources branch 2 times, most recently from 00d5186 to 08ced1a Compare November 28, 2025 11:13

fviernau force-pushed the violation-multiple-license-sources branch 2 times, most recently from b3e3164 to 819ea09 Compare November 28, 2025 12:15

sschuberth reviewed Nov 28, 2025

View reviewed changes

fviernau force-pushed the violation-multiple-license-sources branch from 819ea09 to 8ee235d Compare November 28, 2025 16:10

fviernau requested a review from sschuberth November 28, 2025 16:10

fviernau force-pushed the violation-multiple-license-sources branch 2 times, most recently from be57cb6 to 0bafa29 Compare November 28, 2025 21:57

tsteenbe force-pushed the violation-multiple-license-sources branch from 0bafa29 to 6e5a7b5 Compare December 1, 2025 11:56

fviernau changed the title ~~evaluator: Introduce the option to evaluate a licenseRule for all license source at once~~ evaluator: Introduce the option to evaluate a licenseRule for all license sources at once Dec 1, 2025

fviernau added 4 commits December 4, 2025 10:59

refactor(evaluator)!: Turn licenseSource of violations to a Set

f5932e2

Prepare for combining rule violations that stem from the same license and package, but from different licenses sources. Signed-off-by: Frank Viernau <frank.viernau@gmail.com>

refactor(evaluator)!: Allow multiple license sources in a LicenseRule

d15e242

Prepare for an upcoming change. Signed-off-by: Frank Viernau <frank.viernau@gmail.com>

feat(evaluator): Add API's for backwards compatibility

72fa425

Ensure the previous change is backwards compatible. Signed-off-by: Frank Viernau <frank.viernau@gmail.com>

fviernau force-pushed the violation-multiple-license-sources branch from 6e5a7b5 to 72fa425 Compare December 4, 2025 09:59

evaluator: Introduce the option to evaluate a licenseRule for all license sources at once #11154

Are you sure you want to change the base?

evaluator: Introduce the option to evaluate a licenseRule for all license sources at once #11154

Uh oh!

Conversation

fviernau commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sschuberth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sschuberth Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fviernau commented Nov 27, 2025 •

edited

Loading

codecov bot commented Nov 28, 2025 •

edited

Loading

sschuberth Nov 28, 2025 •

edited

Loading