1 Introduction

Introduction

The test case success rate is a crucial score for measuring test case execution performance. High success rate indicates a robust and dependable test case. On the other hand, a low success rate highlights potential issues or areas of improvement. This signals the need for further investigation and refinement.

Formula to Calculate Success Rate

The system calculates the success Rate of a test case based on the pass rate of the test case.

It helps the QA team determine the effectiveness of the test case and facilitates test planning.

Success Rate = (Number of times the test case version got passed/Number of times the test case version got executed as Passed and Failed) * 100

The status definition for Passed and Failed is defined by the admin on the configuration screen.

Success Rate Instance Level Settings

Super admin can enable it for the QMetry instance from Customization under AI Configuration.

Only the QMetry super admin can access and configure QMetry Intelligence Configuration.

Go to AI Configuration and select Flaky Score and Success Rate.

There are two settings for Flaky Score and Success Rate each.

Allow Project Admins to Enable and Configure the Flaky Score for their project.
Allow Project Admins to Enable and Configure the Success Rate for this project.

Enable “Allow Project Admins to Enable & Configure the Success Rate for this project” settings for Success Rate calculations.

Success Rate Settings Project Level Settings

Project Administrators can configure the Success Rate settings if the option Allow Project Admins to Enable & Configure the Success Rate for their project is enabled in Customization.

Project Admins and users with the Success Rate Modify and Generate permissions can configure the settings for Success Rate.
Users with only “View” rights can view the configuration.

To enable success rate settings, perform the following steps:

1. Go to QMetry Intelligence.

2. Select Success Rate Settings.

3. Open the Configurations tab.

4. Enable Success Rate for the Project.

Once the Success Rate is enabled, the fields for Success Rate Configuration appear editable.

Success Rate Configuration

Administrators can define the scope of test executions for which the Success Rate is calculated. The criteria should be refined to calculate the Success Rate based on the QA reporting methodologies.

AI calculates the Success Rate based on the filtered test execution records according to the criteria. The existing Success Rate resets and recalculated whenever the configuration is modified.

Provide the following details for the Success Rate configuration.

Test Case Executions: Mention the number of the latest test executions of a test case while calculating the success rate. 10 test executions are selected by default. The Test Case Execution value should be from 1 to 1000.
Test Case Versions: Select “Latest Version” or “All Version” while calculating the success rate.
- Latest Version (Default): The test executions of only the latest test case versions is considered.
- All Version: The test executions of all test case versions is considered.
Execution Time Frame: Mention the days to define the time frame in which the test executions were executed. By default, 90 days is selected. If left blank, the system considers all the days.
Release: Select releases to consider test executions for those releases. (Default - All releases).
Cycle: Select cycles to consider test executions for those cycles. (Default - All cycles).
Platform: Select platforms to consider test executions for those platforms. (Default - All platforms).
Build: Select builds to consider the test executions for that particular builds. (Default- All builds).
Test Case Execution Status:
- As Passed: The test execution with these statuses are considered as "Passed" while calculating the Success Rate. The default value is “Passed”.
- As Failed: The test executions with these statuses are considered as "Failed" while calculating the Success Rate. The default values are “Failed” and “Blocked”.

All execution statuses other than selected as passed and failed is considered as Skipped.
Skipped status will not be considered for calculation.

Click Update to save the Success Rate Settings.

To reset the settings, click Reset.

To generate the success rate, click Generate.

The Success Rate can be calculated once in 24 hours for each project.
Once you generate the Success Rate, the Generate button remains disabled for the next 24 hours.

When you hover over the Generate button, it shows the details of the user who generated the Success rate last with the timestamp.

Once you generate the Success Rate, you can view the success rate on the test case list view, test case detail page, test case link screens, linked test cases screen, and test case execution screens.

Displaying Success Rate

The following table interprets the success ratio in accordance with the display of the derived success rate calculation and color code.

Success Rate Range	Success Ratio

Success Rate Range	Success Ratio
Between 81 to 100 %	High
Between 51 to 80 %	Medium
Between 0-50 %	Low

Filtering Test Cases on Success Rate

The test cases can be filtered on Success Rate under the Advanced Filters.

Viewing Success Rate on Test Case List View

The Success Rate is calculated and displayed on the test case list view, based on the Success Rate Settings done in the Configuration.

To view success rate on test case list view, perform the following steps:

Go to the Test Case Module.
Enable the Success Rate column for the list view.

View the Success Rate column with corresponding statistics. You can sort on the column to view test cases with higher success rates.

Click Success Rate to view the traceability report for the test case, which shows the test case is associated with test executions, issues, and execution results. The report helps you analyze the Success Rate further.

The info icon beside the Success Rate column displays the Success Rate details.

If the Success Rate Settings are changed, the existing Success Rate will reset.

View Success Rate on Test Case Detail Page

Success Rate is displayed beside the Test Case Key at the top of the screen.

View Success Rate on Test Case Link Screens

Show or hide the Success Rate column on the Link Test Cases screen.

View Success Rate on Test Execution Screen

To view success rate on test execution screen, perform the following steps:

Go to the Test Execution screen.
Enable Success Rate column.

Test Execution Detail View

If you go further into the Detail View, view the Success Rate count at the top of the page.

Click Success Rate to drill down to view the test executions and defects associated with the test case.

Test Case Success Rate