Test Execution Reports

The Test Execution report helps the testing team, the project manager, and the product owner to understand the status of the testing and the overall stability of the application. The report enables the user to get a detailed understanding of the status of the Test Case within a single or multiple Test Cycle. The report allows the user to view the executions graphically or in tabular format.

Test Execution Reports can be shared and discussed during retrospectives as they contain detailed test execution status. The report can help the organization visualize its current quality and plan QA and Development activities timely.

For the selected Project, the Test Execution Report shows how many tests were run over a period of Time, Sprint, or Version. The report displays the overall progress of test executions.

QMetry allows users to configure and generate reports to retrieve the details with a variety of customization. For further details, you can browse through the https://qmetrysupport.atlassian.net/wiki/spaces/QTM4J/pages/1000931606 and https://qmetrysupport.atlassian.net/wiki/spaces/QTM4J/pages/1000866249.

Note: The report export does not include the Test Execution custom fields in it.

The different options available to generate the Test Execution Report are mentioned below.

Test Case Execution Summary Report

The Test Case Execution report helps the testing team, the project manager, and the product owner understand the status of testing and the stability of the overall application. The report shows the count of test cases based on statuses like passed, blocked, failed, not executed, etc. The report can be grouped on the following mentioned parameters.

• By Project

• By Test Cycle

• By Environment

• By Build

• By Execution Assignee

• By Label

• By Component

• By Test Case Type

• By Requirement

• By Time Frame: The report displays the count of test cycle executions performed in a project(s) during the defined time frame of Daily, Weekly, Monthly, and Yearly. In addition to a specified number of days, you can define a time frame using custom date ranges and operators like =, >, <, <=,>=. The options to select a period depend on the frequency you select to generate the report. The following table displays the maximum duration set to generate the report.

Open

Open

Once the Test Case Execution Report is configured and generated considering By Test Cycle for example.

• Test Case Execution Report can be generated in the bar chart, line chart, pie chart, and table formats.

  • The report can be exported to an Excel file for sharing. Users can export the report using the export button.

  • Test Execution Summary can be exported using the following options:

    1. With steps: The exported report contains all Test Steps for all Test Cases in the selected Test Cycle.

    2. Without steps: The exported report contains only the list of Test Cases in the selected Test Cycle.

The bar graph displays the test cycle summary, test case count, and corresponding percentage on hover.

Line Chart:

The pie chart displays a drop-down menu with values appropriate to the grouping of Test Case Execution Summary report. For example, for Test Case Execution Summary By Test Cycle the pie chart shows the drop-down menu with the test cycles that are covered in the report. The same way, the pie chart report for Test Case Execution Summary By Build will show the drop-down menu with builds that are considered in generating the report.

The tabular view displays the test execution summary in a two-dimensional view that includes the count of the executed test cases and the percentage. Users can click on the Test Cycle Key to view the test cycle details.

Additionally, 2 new filters have been included with DD/MM/YYYY format which enables users to export reports based on “Created” and “Updated” details for Test Assets as shown in the screenshot below:
The example below is for Test Case filtered by “Created On” and “Updated On” for Test Case Execution Summary “By Test Cycle”.

Test Case Execution Overall Effort Summary

The report provides a comparison between the Estimated Time for test case execution vs. Actual Time taken for test case execution. The Test Case Execution Overall Effort Summary Report would help the user to make sure that the tasks are being completed in a set amount of time; through which the user will be able to understand the lag,

Once the Test Case Execution Overall Effort Summary is configured and generated.

• The report can be viewed in three distinct graphs: Bar Chart, Line Chart, and Pie Chart

• Also, a tabular form can be used to represent the data.

• The report can also be exported into an Excel file for sharing. Users can use the Export button for the same.

• The Test Case Execution Overall Effort Summary can be exported using the following options:

  1. With steps: The exported report contains all Test Steps for all Test Cases.

  2. Without steps: The exported report contains only the list of Test Cases.

Test Case Execution Effort Summary By Execution Assignee

The report displays the Execution Assignee-wise summary of the Estimated Time vs. Actual Time of test case execution. It shows the actual time taken by the Execution Assignee to whom the Test Case was assigned for execution. If the user wishes to get the overall efficiency in terms of effort put on by an individual assignee for a given execution, the user can make use of the Test Case Execution Effort Summary By Assignee.

Once the Test Case Execution Effort Summary By Execution Assignee is configured and generated.

• The report can be exported into an Excel file for sharing. Users can use the Export button for the same.

• The Test Case Execution Effort Summary By Execution Assignee can be exported using the following options:

  1. With steps: The exported report contains all Test Steps based on the execution assignee.

  2. Without steps: The exported report contains only the list of Test Cases based on the execution assignee.

Target vs Actual Execution

The Target vs Actual Execution report enables users to see the execution progress or target completion date for the target execution.

Target vs Actual Execution report under the Test Execution Reports helps QA Managers understand the current test execution progress against the targeted execution to achieve the timeline. You can generate reports for the following calculation approaches:

• Target Completion Date: For the specified duration, the report displays the Planned Test Execution Rate, Current Test Execution Rate, and Required Test Execution Rate to meet the release timelines. The report displays a Line Chart that indicates the execution progress during a particular sprint/period. It helps leads/managers review the current progress of test execution against the target test execution and make decisions regarding resource allocation/task distribution to achieve the target.

• Execution Velocity: The Execution Velocity helps QA Managers measure how fast the test executions are performed compared to the set target. For the specified start date and per-day test execution velocity, the report displays the Planned Completion Date and current Test Execution Velocity. It calculates the Test Execution completion date based on the Current Test Execution Velocity.

Generate Report

Follow the steps mentioned below to configure the report and generate it.

Step 1. Define Scope

The test execution scope is defined through the selection of the Project.

Note: The Target Vs Actual Execution report can be generated only for a single project.

You can apply filters to select the required test cycles to generate the report. You can apply filters like Test Cycle Keys, Folder Path, Fix Version, Sprint, Component, Labels, Priority, Assignee/Reporter, Planned Start Date/End Date, Test Cycle Custom fields, and Story Filter JQL to consider test cycle directly linked to Stories.

To select the required test executions to generate the report, you can apply filters like Execution Assignee, Environments, Builds, Execution Assignee, Executed By, Executed On, and Latest/All Test Executions.

Step 2. Additional Settings

• View By: You can view the report for the frequency you select from (a)Daily (default) (b) Weekly (c) Monthly

  • Days to be skipped: If you select View By “Daily”, you get an option to skip the days the data of which you want to omit from the calculation.

• Calculation Approach: There are two Calculation Methods.

  • (A) Target Completion Date:

    • Execution Start Date and End Date

    • Select Execution Statuses that are considered as executed. You can select multiple Execution Statuses from the dropdown.

  • (B) Execution Velocity:

    • Execution Start Date

    • Mention the Test Execution Velocity per day. This is your planned execution velocity.

    • Select Execution Statuses that are considered as executed. You can select multiple Execution Statuses from the dropdown.

Click on the Generate button to generate the report.

You can opt to view the line chart either in Burnup or Burndown pattern. The chart is generated in the Burndown pattern by default.

View Reports

You can generate and view the following reports:

(A) Target Completion Date

(B) Execution Velocity

The report calculates the average number of executions to be completed daily based on the total number of days used and the total number of executions.

• Total Executions Count: Total test executions to be run.

• Remaining Executions Count: Total test executions to be run - Test Executions already run

• Planned Test Execution Rate: It is calculated based on the Start and End date (total number of days) and planned test executions.

• Current Test Execution Rate: This is calculated based on the count of completed test executions and the total number of days.

• Required Test Execution Rate: Test Execution Rate required to complete the test execution as per the Execution End Date. It is calculated based on the remaining executions and total number of days.

• Execution Start Date: It is mentioned during the configuration of the report.

• Execution End Date: It is mentioned during the configuration of the report.

• View By: Selected during the configuration of the report.

• Calculation Approach: It is selected during the configuration of the report.

A cumulative line chart is generated that displays the count of Planned Test Executions and Actual Test Executions. The cumulative Daily Count (i.e. The average count of test execution runs to be completed on each date as per the calculation) is displayed on the Y-axis and Dates are displayed on the X-axis.

• Planned: It is calculated based on the Start and End date (total count of days).

Total Count of Planned Test Execution/Total Number of Days

• Actual: It is calculated based on the daily total execution count (as per the executed date) that matches selected execution statuses.

Burndown Chart (default)

Burnup Chart

Velocity is a measure of the progress of test execution by the QA team. It refers to the rate at which the QA team consistently runs the test executions. This predictive metric helps the QA Manager to determine the Test Execution Completion Date by which the defined scope could be achieved.

The report calculates the Test Execution Completion Date based on the Remaining Executions Count and Current Test Execution Velocity.

A cumulative line chart is generated that displays the count of Planned Test Executions and Actual Test Executions.

• Total Executions Count: Total test executions to be run.

• Remaining Executions Count: Total test executions to be run - Test Executions already run

• Planned Test Execution Velocity: The Test Execution Velocity was mentioned during the configuration of the report.

• Current Test Execution Velocity: It is calculated based on the actual test executions/Number of Days.

• Test Execution will be completed on: The date is calculated based on the count of remaining test executions and Current Test Execution Velocity.

• Execution Start Date: The date is mentioned during the configuration of the report.

• Planned Completion Date: The date is calculated based on the Test Execution Velocity mentioned during the configuration of the report.

• View By: It is selected during the configuration of the report.

• Calculation Approach: It is selected during the configuration of the report.

A cumulative line chart is generated that displays the count of Planned Test Execution Velocity and Actual Test Execution Velocity.

The cumulative Daily Count is displayed on the Y-axis and Dates are displayed on the X-axis.

• Planned: It is calculated based on the user-provided daily execution velocity. The following formula is applied for calculation.

• Actual: It is calculated based on the daily total execution count (as per the executed date) that matches selected execution statuses.

The cumulative line chart is generated accordingly and the end date is calculated based on speed.

Burndown Chart (default)

Burnup Chart

You can view the Tabular View of the report through either of the ways:

(A) By clicking on the Tabular View option

(B) By expanding the Report Tabular View section below the Line Chart