1 Introduction
2 Success Rate Instance Level Settings
3 Success Rate Settings Project Level Settings
4 Display of Success Rate
5 Filter Test Cases on Success Rate
6 View Success Rate on Test Case List View
7 View Success Rate on Test Case Detail Page
8 View Success Rate on Test Case Link Screens
9 View Success Rate on Test Execution Screen

Introduction

Introducing the success rate for test cases, calculated based on their execution history, holds significant importance in the testing process. The success rate provides QA teams with a quantitative measure of the reliability and effectiveness of the test cases. The test case success rate is a crucial score for measuring test case execution performance, as a high success rate indicates a robust and dependable test case. On the other hand, a low success rate highlights potential issues or areas of improvement, signaling the need for further investigation and refinement.

Formula to Calculate Success Rate

The Success Rate of a test case is calculated based on the pass rate of the test case. It helps the QA team determine the effectiveness of the test case and facilitates test planning.

Success Rate = (Number of times the test case version got passed/Number of times the test case version got executed as Passed and Failed) * 100

The Status definition for Passed and Failed is defined by the admin on the configuration screen.

Success Rate Instance Level Settings

These are the settings for the QTM instance. To activate the AI feature for QMetry users, the QTM super administrator needs to enable it for the QMetry instance from Customization > General Settings & Audit > AI Configuration.

→ Required Permission: Only the QMetry super admin can access and configure QMetry Intelligence Configuration.

AI Configuration > Flaky Score and Success Rate Configuration (Default: Disabled)

There are two settings for Flaky Score and Success Rate each.

(A) Allow Project Admins to Enable & Configure the Flaky Score for their project.
(B) Allow Project Admins to Enable & Configure the Success Rate for this project.

Enable the “Allow Project Admins to Enable & Configure the Success Rate for this project” settings for Success Rate calculations.

Success Rate Settings Project Level Settings

→ Prerequisite: Project Administrators can configure the Success Rate settings if the option “Allow Project Admins to Enable & Configure the Success Rate for their project.” is enabled in Customization > General Settings & Audit > AI Configuration.

→ Required Permissions

Project Admins and users with the Success Rate “Modify” and “Generate” permissions can configure the settings for Success Rate. Users with only “View” rights can view the configuration.

Steps

1. Go to QMetry Intelligence.

2. Select the Success Rate Settings option.

3. Open the Configurations tab.

4. Enable Success Rate for the Project.

Once the Success Rate is enabled, the fields for Success Rate Configuration appear editable.

Success Rate Configuration

The settings on this page allow Administrators to define the scope of test executions for which the Success Rate will be calculated. The criteria should be refined to calculate the Success Rate based on the QA reporting methodologies.

AI will calculate the Success Rate based on the filtered test execution records according to the criteria. The existing Success Rate will be reset and recalculated whenever the configuration is modified.

Provide the following details for the Success Rate configuration.

Test Case Executions: Mention the number of the latest test executions of a test case you want to consider while calculating the success rate. 10 test executions are selected by default. The Test Case Execution value should be from 1 to 1000.
Test Case Versions: Select the test case versions either “Latest Version” or “All Version” while calculating the success rate. The “Latest Version” is selected by default.
- Latest Version: The test executions of only the latest test case versions will be considered while calculating the Success Rate.
- All Version: The test executions of all test case versions will be considered while calculating the Success Rate.
Execution Time Frame: Mention the days to define the timeframe in which the test executions were executed in the last specified days. 90 days are selected by default, which indicates that the test executions carried out during the previous 90 days will be considered while calculating the Success Rate. If left blank, it will be considered as all the days.
Release: Select Release(s) to consider test executions for those releases. “All” releases are selected by default.
Cycle: Select Cycle(s) to consider test executions for those cycles. “All” cycles are selected by default.
Platform: Select Platform(s) to consider test executions for those platforms. “All” platforms are selected by default.
Build: Select build(s) to consider the test executions about that particular build(s). “All” builds are selected by default.
Test Case Execution Status (All execution statuses other than selected as passed and failed will be considered as Skipped. Skipped status will not be considered for calculation).
- As Passed: The test execution with these statuses will be considered as "Passed" while calculating the Success Rate. The default value is “Passed”.
- As Failed: The test executions with these statuses will be considered as "Failed" while calculating the Success Rate. The default values are “Failed” and “Blocked”.

Click Update to save the Success Rate Settings.

To reset the settings, click on the Reset button.

To generate the success rate, click on the Generate button.

The success message pops up.

Note: The Success Rate can be calculated once in 24 hours for each project. Once you generate the Success Rate, the Generate button will remain disabled for the next 24 hours.

When you hover over the Generate button, it shows the details of the user who generated the Success rate last with the timestamp.

Once you generate the Success Rate on the Success Rate Settings screen, you can view the success rate on the test case list view, test case detail page, test case link screens, linked test cases screen, and test case execution screens.

Display of Success Rate

The following table interprets the success ratio in accordance with the display of the derived success rate calculation and color code.

Success Rate Range	Success Ratio

Success Rate Range	Success Ratio
Between 81 to 100 %	High
Between 51 to 80 %	Medium
Between 0-50 %	Low

Filter Test Cases on Success Rate

The test cases can be filtered on Success Rate. You can find the Success Rate under the Advanced Filters.

View Success Rate on Test Case List View

As per the Success Rate Settings done in the Configuration, the Success Rate is calculated and displayed on the test case list view.

Go to the Test Case module and make sure you make the Success Rate column visible for the list view.

Success Rate: It is calculated based on the pass rate of the test case.

You can see the Success Rate column with corresponding statistics. You can sort on the column to view test cases with higher success rates.

Click on the Success Rate to view the traceability report for the test case, which shows the test case is associated with test executions, issues, and execution results. The report helps you analyze the Success Rate further.