/
Flaky Test Detection

Flaky Test Detection

 

Introduction

QTM4J has introduced an AI-powered, innovative approach to assess the flakiness of test cases by calculating a "Flaky Score" derived from their execution history; this insightful feature empowers testers an ability to identify test cases whose future execution status is non-deterministic. A flaky test case refers to one that exhibits non-deterministic behavior when executed repeatedly within the same code and environment, resulting in intermittent successes and failures. The crucial first step towards gaining control over flaky tests is identifying them. Flaky tests have the potential to slow down testing pipelines and erode confidence in testing processes. Today determining test case flakiness requires a manual comparison of test results from multiple runs, which is time-consuming. However, with QMetry Intelligence, this process is now automated, saving valuable time and effort for testers and developers.

QA Managers can define and configure settings according to their specific testing processes for calculating the flaky score, ensuring its relevance to their testing methodologies.

For Example,

The following table shows the execution results of the test cases executed multiple times.

Test Case Name

Test 1

Test 2

Test 3

Test 4

Test 5

Test 6

Test 7

Test 8

Flaky or Non-flaky?

Test Case Name

Test 1

Test 2

Test 3

Test 4

Test 5

Test 6

Test 7

Test 8

Flaky or Non-flaky?

Test Case A

Pass

Pass

Pass

Pass

Pass

Pass

Pass

Pass

Non-flaky

Test Case B

Fail

Fail

Fail

Fail

Fail

Fail

Fail

Fail

Non-flaky

Test Case C

Pass

Pass

Fail

Fail

Pass

Fail

Pass

Fail

Flaky

 

Use Cases:

  • QA Managers/Testers get to view the risk probability associated with the test cases under test.

  • The flaky score on the execution screen shows the tester the probability of risk while executing the test case.

Flaky Score is calculated only for the executed test cases and is determined based on the latest X number of executions. The Flaky Score ranges between 0 (Not Flaky) and 1 (Flaky).

Required Permissions:

Users should have the following QMetry Intelligence permissions to configure and generate Flaky Score:

  • Modify Flaky and Success Rate Configuration

  • Calculate Flaky and Success Rate Score

Notes:

  • The final execution status assigned to the test case will only be considered.

  • The Flaky Score is calculated only for the test cycles of the same project.

Flaky Score Settings

Steps:

1. Go to Apps > QMetry > QMetry Intelligence.

2. Select the Flaky Score Settings option and go to the Configuration tab.

3. Provide the following details for flaky score rate configuration:

Display Flaky Score Settings:

  • The feature provides users with the option to toggle the display of the flaky score. If the feature is enabled, it allows configuration for laky Score computation. QA managers should configure and refine the criteria to calculate Flaky Score based on their QA reporting methodologies. 

Users can then opt to show/hide the Flaky Score column on the test case list view, test cycle > test cases tab, test execution screen, and Test Case/Acceptance Criteria section on the Jira story page. The Flaky Score is also displayed on the test case detail page.

image-20240812-075005.png

Flaky Score Configuration:

  • Test Case Executions: Mention the number of the latest test executions for a test case you want to consider while calculating the flaky score. 10 test executions are selected by default. The Test Case Execution value should be from 1 to 1000.

  • Test Case Versions: Select the executions for either the “Latest” or “All” test case versions to consider while calculating the flaky score. The “Latest” test case version is selected by default.

    • Latest: The test executions of only the latest test case versions will be considered while calculating the flaky score.

    • All: The latest test executions of all test case versions will be considered while calculating the flaky score.

  • Environment: Select environment(s) to consider the test execution run against that particular environment(s). “All” environments are selected by default.

  • Build: Select build(s) to consider the test executions pertaining to that particular build(s). “All” builds are selected by default.

  • Fix Version: Select Fix Version(s) to consider the test executions for the test cycles with the selected Fix Version(s). “All” Fix Versions are selected by default.

  • Sprint: Select sprint(s) to consider the test executions for the test cycles covered in that particular sprint(s). “All” sprints are selected by default.

  • Execution Timeframe (days): Mention the days to define the timeframe in which the test executions were executed in the last specified days. 90 days are selected by default, which indicates the test executions carried out in the last 90 days will be considered while calculating the flaky score. The Execution Timeframe value should be from 1 to 365 days.

Set Test Case Status

  • As Passed: Select the status(es) to define the test case status as “Passed”. You can select multiple statuses from the list.

  • As Failed: Select the status(es) to define the test case status as “Failed”. You can select multiple statuses from the list.

 

Click Save to save the Flaky Score Settings.

To reset the settings, click on the Reset to Default button.

To generate the flaky score, click on the Generate button.

image-20240810-085332.png

You can view the progress of the background process in the Notifications section.