Skip to content

fix(specs): abtests metadata mean #5121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 16, 2025
Merged

fix(specs): abtests metadata mean #5121

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 13 additions & 7 deletions specs/abtesting-v3/common/schemas/Variant.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,17 +57,17 @@ metricResult:
type: number
format: double
description: |
The upper bound of the 95% confidence interval for the metric value. The confidence interval is calculated using
The upper bound of the 95% confidence interval for the metric value. The confidence interval is calculated using
either the relative ratio or relative difference between the metric values for the control and the variant.
Relative ratio is used for metrics that are ratios (e.g., click-through rate, conversion rate),
Relative ratio is used for metrics that are ratios (e.g., click-through rate, conversion rate),
while relative difference is used for continuous metrics (e.g., revenue).
valueCILow:
type: number
format: double
description: |
The lower bound of the 95% confidence interval for the metric value. The confidence interval is calculated using
The lower bound of the 95% confidence interval for the metric value. The confidence interval is calculated using
either the relative ratio or relative difference between the metric values for the control and the variant.
Relative ratio is used for metrics that are ratios (e.g., click-through rate, conversion rate),
Relative ratio is used for metrics that are ratios (e.g., click-through rate, conversion rate),
while relative difference is used for continuous metrics (e.g., revenue).
pValue:
type: number
Expand All @@ -83,7 +83,7 @@ metricResult:
format: double
description: |
The value that was computed during error correction. It is used to determine significance of the metric pValue.
The critical value is calculated using Bonferroni or Benjamini-Hochberg corrections, based on the given
The critical value is calculated using Bonferroni or Benjamini-Hochberg corrections, based on the given
configuration during the A/B test creation.
significant:
type: boolean
Expand Down Expand Up @@ -131,9 +131,15 @@ metricMetadata:
type: number
format: double
description: |
Only present in case the metric is 'revenue'.
Only present in case the metric is 'revenue'.
It is the amount exceeding the 95th percentile of global revenue transactions involved in the AB Test. This amount is not considered when calculating statistical significance.
It is tied to a per revenue-currency pair contrary to other
It is tied to a per revenue-currency pair contrary to other
global filter effects (such as outliers and empty search count).
mean:
type: number
format: double
description: Mean value for this metric.
example: 53.7
example:
winsorizedValue: 888.80
mean: 53.7
Loading