Skip to content

Olek.rudyy/test #33193

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
TovRudyy wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

Olek.rudyy/test #33193

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
45e7cf3
udpate docs
TovRudyy Dec 5, 2025
75214ff
fix
TovRudyy Dec 5, 2025
ff6a0a5
update product name
TovRudyy Dec 5, 2025
a7b2419
refactor table in html
TovRudyy Dec 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 47 additions & 11 deletions content/en/tests/flaky_management/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: Flaky Test Management
title: Flaky Tests Management
description: Track, triage, and manage flaky tests.
further_reading:
- link: "/continuous_integration/tests/"
Expand All @@ -19,7 +19,7 @@ further_reading:

## Overview

The [Flaky Test Management][1] page provides a centralized view to track, triage, and remediate flaky tests across your organization. You can view every test's status along with key impact metrics like number of pipeline failures, CI time wasted, and failure rate.
The [Flaky Tests Management][1] page provides a centralized view to track, triage, and remediate flaky tests across your organization. You can view every test's status along with key impact metrics like number of pipeline failures, CI time wasted, and failure rate.

From this UI, you can act on flaky tests to mitigate their impact. Quarantine or disable problematic tests to keep known flakes from breaking builds, and create cases and Jira issues to track work toward fixes.

Expand Down Expand Up @@ -48,12 +48,48 @@ Configure automated Flaky Test Policies to govern how flaky tests are handled in

3. Use the toggles to enable specific automated actions, and use automation rules to further customize how tests get quarantined, disabled, or retried:

| Action | Description |
| ---- | ---- |
| **Quarantine** | Toggle to allow flaky tests to be quarantined for this repository. Customize automation rules based on: <li>Time: Quarantine a test if its status is `Active` for a specified number of days. <li>Branch: Quarantine an `Active` test if it flakes in one or more specified branches.|
| **Disable** | Toggle to allow flaky tests to be disabled for this repository. You may want to do this after quarantining or to protect specific branches from flakiness. Customize automation rules based on: <li>Status and time: Disable a test if it has a specified status for a specified number of days. <li>Branch: Disable an `Active` or `Quarantined` test if it flakes in one or more specified branches. |
| **Attempt&nbsp;to&nbsp;Fix** | When you attempt to fix a flaky test, automatically retry the test a specified number of times on the commit containing the fix. |
| **Fixed** | If a flaky test no longer flakes for 30 days, it is automatically moved to Fixed status. This automation is default behavior and can't be customized. |
<table>
<thead>
<tr>
<th>Action</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Quarantine</strong></td>
<td>
<p>Toggle to allow flaky tests to be quarantined for this repository.</p>
<p>Customize automation rules based on:</p>
<ul>
<li><strong>Time</strong>: Quarantine a test if its status is <code>Active</code> for a specified number of days. The rule is triggered every day at 12:15 UTC.</li>
<li><strong>Branch</strong>: Quarantine an <code>Active</code> test if it flakes in one or more specified branches.</li>
<li><strong>Failure rate</strong>: Quarantine an <code>Active</code> test if its failure rate over the last 7 days is greater or equal to the specified threshold. The rule is triggered every 15 minutes.</li>
</ul>
</td>
</tr>
<tr>
<td><strong>Disable</strong></td>
<td>
<p>Toggle to allow flaky tests to be disabled for this repository. You may want to do this after quarantining or to protect specific branches from flakiness.</p>
<p>Customize automation rules based on:</p>
<ul>
<li><strong>Status and time</strong>: Disable a test if it has a specified status for a specified number of days. The rule is triggered every day at 12:30 UTC.</li>
<li><strong>Branch</strong>: Disable an <code>Active</code> or <code>Quarantined</code> test if it flakes in one or more specified branches.</li>
<li><strong>Failure rate</strong>: Disable an <code>Active</code> or <code>Quarantined</code> test if its failure rate over the last 7 days is greater or equal to the specified threshold. The rule is triggered every 15 minutes.</li>
</ul>
</td>
</tr>
<tr>
<td><strong>Attempt&nbsp;to&nbsp;Fix</strong></td>
<td>When you attempt to fix a flaky test, automatically retry the test a specified number of times on the commit containing the fix.</td>
</tr>
<tr>
<td><strong>Fixed</strong></td>
<td>If a flaky test no longer flakes for 30 days, it is automatically moved to Fixed status. This automation is default behavior and can't be customized.</td>
</tr>
</tbody>
</table>

## Track evolution of flaky tests

Expand Down Expand Up @@ -83,7 +119,7 @@ For any flaky test, you can create a case and use [Case Management][4] to track

When you fix a flaky test, Test Optimization's remediation flow can confirm the fix by retrying the test multiple times. If successful, the test's status is automatically updated to `Fixed`. To enable the remediation flow:

1. For the test you are fixing, click **Link commit to Flaky Test fix** in the Flaky Test Management UI.
1. For the test you are fixing, click **Link commit to Flaky Test fix** in the Flaky Tests Management UI.
1. Copy the unique flaky test key that is displayed (for example, `DD_ABC123`).
1. Include the test key in your Git commit title or message for the fix (for example, `git commit -m "DD_ABC123"`).
1. When Datadog detects the test key in your commit, it automatically triggers the remediation flow for that test:
Expand All @@ -108,7 +144,7 @@ To enable AI-powered flaky test fixes, enable Bits AI Dev Agent for Test Optimiz

## AI-powered flaky test categorization

Flaky Test Management uses AI to automatically assign a root cause category to each flaky test based on execution patterns and error signals. This helps you filter, triage, and prioritize flaky tests more effectively.
Flaky Tests Management uses AI to automatically assign a root cause category to each flaky test based on execution patterns and error signals. This helps you filter, triage, and prioritize flaky tests more effectively.

<div class="alert alert-info">A test must have at least one failed execution that includes both <code>@error.message</code> and <code>@error.stack</code> tags to be eligible for categorization. If the test was recently detected, categorization may take several minutes to complete.</div>

Expand Down Expand Up @@ -141,7 +177,7 @@ Notifications are not sent immediately; they are batched every few minutes to re

## Compatibility

To use Flaky Test Management features, you must use Datadog's native instrumentation for your test framework. The table below outlines the minimum versions of each Datadog tracing library required to quarantine, disable, and attempt to fix flaky tests. Click a language name for setup information:
To use Flaky Tests Management features, you must use Datadog's native instrumentation for your test framework. The table below outlines the minimum versions of each Datadog tracing library required to quarantine, disable, and attempt to fix flaky tests. Click a language name for setup information:

| Language | Quarantine & Disable | Attempt to fix |
| --------------- | ----------------------------- | ---------------------------- |
Expand Down
Loading