Introduction

Introducing the success rate for test cases, calculated based on their execution history, holds significant importance in the testing process. The success rate provides QA teams with a quantitative measure of the reliability and effectiveness of the test cases. The test case success rate is a crucial score for measuring test case execution performance, as a high success rate indicates a robust and dependable test case. On the other hand, a low success rate highlights potential issues or areas of improvement, signaling the need for further investigation and refinement. 

Formula to Calculate Success Rate

The Success Rate of a test case is calculated based on the pass rate of the test case. It helps the QA team determine the effectiveness of the test case and facilitates test planning.

(Number of times the test case version was passed/Number of times the test case version was executed as Passed and Failed) * 100

The Status definition for Passed and Failed is defined by the Admin from the configuration screen.

Required Permissions:

Users should have the following QMetry Intelligence permissions to configure and generate Success Rate:

• Modify Flaky and Success Rate Configuration

• Calculate Flaky and Success Rate Score

Success Rate Settings

Steps:

1. Go to Apps > QMetry > QMetry Intelligence.

2. Select the Success Rate Settings option and go to the Configuration tab.

3. Provide the following details for success rate configuration:

Display Success Rate Settings:

• The feature provides users with the option to toggle the display of the success rate. If the feature is enabled, you can configure Success Rate computation. Users can then opt to show/hide the Success Rate column on the test case list view, test cycle > test cases tab, test execution screen and Test Case/Acceptance Criteria section in Jira story page. The Success Rate is also displayed on the test case detail page.

Open

Success Rate Configuration:

• Test Case Executions: Mention the number of the latest test executions of a test case you want to consider while calculating the success rate. 10 test executions are selected by default.

• Test Case Versions: Select the test case versions either “Latest” or “All” to consider while calculating the success rate. The “Latest” test case version is selected by default.

  • Latest: The test executions of only the latest test case versions will be considered while calculating the success rate.

  • All: The test executions of all test case versions will be considered while calculating the success rate.

• Fix Versions: Select Fix Version(s) to consider the test executions for the test cycles with the selected Fix Version(s). “All” Fix Versions are selected by default.

Set Test Case Status

• As Passed: Select the test execution status(es) to define the test case status as “Passed”. You can select multiple statuses from the list. “Pass” status is selected by default.

• As Failed: Select the test execution status(es) to define the test case status as “Failed”. You can select multiple statuses from the list. “Blocked” and “Fail” statuses are selected by default.

Click Save to save the Success Rate Settings.

To reset the settings, click on the Reset to Default button.

Open

To generate the success rate, click on the Generate button.

When you hover over the Generate button, it shows the details of the user who generated the Success rate last with the timestamp.

Once you generate the Success Rate on the Success Rate Settings screen, you can view the success rate on the test case list view, test case detail page, test cycle > test cases tab, test execution screen, and Test Case/Acceptance Criteria section in the story page.

Display of Success Rate

The following table interprets the success ratio in accordance with the display of the derived success rate calculation and color code.

View Success Rate on Test Case List View

As per the Success Rate Settings done in the Configuration, the Success Rate is calculated and displayed on the test case list view.

Go to the Test Case module and make sure you make the Success Rate column visible for the list view.

• Success Rate: It is calculated based on the pass rate of the test case.

You can see the Success Rate column with corresponding statistics. You can sort on the column to view test cases with higher success rates.

The info icon beside the Success Rate column displays the details of when the Success Rate was last generated and by whom.

If the Success Rate Settings are changed, the existing Success Rate will get reset.

View Success Rate on Test Case Detail Page

The generated Success Rate is displayed beside the Test Case Key at the top of the screen.

View Success Rate on Test Cycle > Test Cases tab

1. Go to the Test Cycle module.

2. Open the test cycle details page and go to the Test Cases tab.

3. Make the Success Rate column visible on the screen.

The Test Cases tab displays the Success Rate statistics.

View Success Rate on Test Execution Screen

You can view the Success Rate statistics just adjacent to the test case key at the top.

Grid View

List View

Make the Success Rate column visible on the screen.

View Success Rate in Story

The success rate of the linked test cases is displayed in the Story. It helps QA Teams to determine the stability of the story by looking at the linked test case execution performance.

Go to the Story > Test Case/Acceptance Criteria and make sure you make the Success Rate column visible.

You can see the Success Rate column with relevant statistics.