Risk Based Testing: Auto Calculation of RPN and Extent Of Testing

Note: The Risk Based Testing feature is only available with the Advanced Features pack of QMetry.

Introduction

Users can define and measure Risks. Associate them with Requirement and Test Cases and generate RBT reports.

RBT - Risk-based testing (RBT) is a software testing type that is based on the probability of risk. It involves assessing the risk based on software complexity, the criticality of business, frequency of use, possible areas with defects, and areas of the system that have changed the most. Risk-based testing prioritizes the testing of features and functions of the software application which have the most likely exposure to risks and therefore most likely to have defects.

Configuration

Required Permissions:

  • Users must possess the Project “View” rights in order to access and view the "Enable Risk Based Analysis" settings.

  • Users must possess the Project “Modify” rights in order to edit the "Enable Risk Based Analysis" Settings.

Steps

1. Go to Projects > Project/Release/Cycle.

2. Open the General Settings tab.

3. You will see the Risk Based Analysis section on the screen.

  • Enable Risk Based Analysis in the Project for the below entities: You can enable this option to implement the Risk Based Analysis for Requirements and Test Cases. You can select both the modules or either of them.

On: It activates the "Risk Based Analysis" section within the selected entities (i.e., requirements, test cases). This section automatically includes six predefined risk fields.

→ Off: It disables the "Risk Based Analysis" section and its associated six risk fields within the selected entities (i.e., requirements, test cases). Any existing data within these fields will be preserved and not automatically deleted. 

image-20240522-140116.png

If "Risk Based Analysis " is enabled:

  • “Risk Based Analysis" fields are displayed for unarchived test cases, Data Parameterized test cases, and Exploratory test cases.

  • If the entities are being copied in the same project: the values will be copied.

  • If the entities are being copied across projects: values will be copied only when RBT is enabled.

  • Projects with Jira integration: The following fields will be displayed on the Test Case detail page in Jira.

    • Risk Type 

    • Risk Category

    • Likelihood

    • Impact

    • Risk Priority Number (RPN) 

    • Extent of Testing

 

 If "Risk Based Analysis" is disabled: The values are retained. They will only be hidden. It does not get deleted on its own. If the option is enabled again, the RBT values will be available to use.

Impact across QTM when RBT is enabled

Manage Fields

A) The following fields are auto-created as System Fields under the Customization > Manage Fields section. You can make these fields mandatory. You can add new values for the fields and archive, delete, and customize the values.

  • Risk Type 

  • Risk Category

  • Likelihood

  • Impact

Lists

B) The following Lists are auto-created under the Customization > Lists section. You can edit existing values and add new values to the list.

  • Risk Likelihood: The list is created with the values following values.

Value

Description

Value

Description

Value

Description

Value

Description

1

Very High

3.5

Between Medium and Low

1.5

Between VeryvHigh and High

4

Low

2

High

4.5

Between Low and Least

2.5

Between High and Medium

5

Least

3

Medium

 

 

  • Risk Impact: The list is created with the following values.

Value

Description

Value

Description

Value

Description

Value

Description

1

Very High

3.5

Between Medium and Low

1.5

Between VeryvHigh and High

4

Low

2

High

4.5

Between Low and Least

2.5

Between High and Medium

5

Least

3

Medium

 

 

  • Risk Type: The list is created with the values Project, Product, Process, and None.

  • Extent of Testing: The list is created with the following values.   

E. T. Range (RPN Value)

Accepted format >{Number}-{Number}

Value

E. T. Range (RPN Value)

Accepted format >{Number}-{Number}

Value

E. T. Range (RPN Value)

Accepted format >{Number}-{Number}

Value

E. T. Range (RPN Value)

Accepted format >{Number}-{Number}

Value

>0 - 5

Extensive

>20 - 25

Report Bug Only

>5 - 10

Broad

>25 - 35

None [Custom Defined by Project Admin]

>10 - 15

Cursory

Out of range

Not Defined

>15 - 20

Opportunity

 

 

  • Risk Category: The list is created with the following values. 

Risk Category

Description

Competitive Inferiority 

Failures to match competing systems in quality.

Data Quality

Failures in processing, storing, or retrieving data.

Date and Time Handling

Failures in date-based and/or time-based inputs/outputs, calculations, and event handling.

Disaster Handling and Recovery

Failure to degrade gracefully in the face of catastrophic incidents and/or failure to recover properly from such incidents.

Error Handling and Recovery

Failures due to bad inputs, beyond peak, or other illegal conditions (i.e., knock-on effects of deliberately inflicted errors).

Functionality

Failures that cause specific features not to work.

Installation, Setup, Upgrade, and Migration

Failures that prevent or impede deploying the system, and migrating data to new versions, including unwanted side-effects (e.g., installing additional, unwelcome, unintended software such as spyware, malware, etc.).

Interoperability

Failures occur when major components, subsystems, or related systems interact.

Load, Capacity, and Volume

Failures in scaling of system to expected peak concurrent usage levels.

Localization

Failures in specific localities, including languages, messages, taxes and finances, operational issues, and time zones

Networked and Distributed

Failure to handle networked/distributed operation, including latency, delays, lost packet/connectivity, and unavailable resources.

Operations and Maintenance

Failures that endanger continuing operation, including backup/restore processes.

Packaging/Fulfillment

Failures associated with the packaging and/or delivery of the system or product.

Performance

Failures to perform as required under expected loads.

Portability, Configuration, and Compatibility

Failures specific to different supported platforms, supported configurations, configuration problems, and/or cohabitation with other software/systems.

Reliability, Availability, and Stability

Failures to meet reasonable expectations of availability and mean-time-between-failure.

Security/Privacy

Failures to protect the system and secure data from fraudulent or malicious misuse.

Standards Compliance

Failure to conform to mandatory standards, company standards, and/or applicable voluntary standards.

States and Transactions

Failure to properly respond to sequences of events or particular transactions.

Usability

Failures arise from aspects of the system that make players or other users feel ineffective, inefficient, or dissatisfied while using the system.

User Interface

Failures where incorrect information is presented directly to users.

Requirements Module

The Risk Based Analysis section appears on the create and edit screens of requirements. The section includes the following fields.

  • Risk Type: Select from Project, Product, Process, and None. 

  • Risk Category: Select the appropriate value to categorize the risk.

  • Likelihood: Select one from the values 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, and 5 on the list.

  • Impact: The lookup list contains values 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, and 5.

  • Risk Priority Number: The field value is auto-calculated based on the values entered for the fields above.

RPN= Likelihood x Impact

  • Extent of Testing: The value of ET is auto-calculated based on the RPN value range. Refer to the Extent of Testing list for more details.

image-20240503-091632.png

Test Cases Module

The Risk Based Analysis section appears on test cases' create and edit screens. The section includes the following fields.

  • Risk Type: Select from Project, Product, Process, and None. 

  • Risk Category: Select the appropriate value to categorize the risk.

  • Likelihood: Select one from the values 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, and 5 on the list.

  • Impact: The lookup list contains values 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, and 5.

  • Risk Priority Number: The field value is auto-calculated based on the values entered for the fields above.

Risk Priority Number (RPN) = Likelihood x Impact

  • Extent of Testing: The value of ET is auto-calculated based on the RPN value range. Refer to the Extent of Testing list for more details.

Test Case Created from Requirement

When a test case is created from the requirement, the fields related to the Risk Based Analysis will be auto-populated from the requirement.

Tabs/Screens on which RBT Fields will be visible

The Risk Based Analysis related columns are displayed on the following screens. The Filters section also includes the fields that let you filter the records using the RBT fields.

  • Test Case > Requirements tab

  • Test Case > Requirement tab > Link Requirements 

  • Test Case Bulk Operations

  • Requirements > Test Cases tab

  • Requirement Bulk Operations

  • Test Suite > Test Cases tab

 

  • Issues > Requirements tab

 

  • Issues > Requirement tab > Link Requirements

Layout Management

The fields related to Risk Based Analysis are available in the Requirements and Test Cases module in the Layout Management section of Customization.

You can not move fields under Risk Based Analysis section to other sections. You can not hide these fields. The fields should be placed under the Risk Based Analysis section only.

Requirements and Test Case Import and Export

  • Import: The values of all Risk Based Analysis fields are imported during the import of requirements and test cases.

  • Export:

    • When the requirement and test case details are exported to XLXS and CSV, all the field values in the Risk Based Testing section will be exported along with other details.

    • When the requirement and test case details are exported to PDF and Word, the Risk Based Testing section will be exported.

    • You can Bulk Export Test Case Details with all Risk Based Analysis fields from the Test Suite > Test Cases tab.

Visual Reports

Users can generate tables and Add gadgets using Risk Fields.

The fields related to Risk Based Testing are available for requirements and test cases to generate the reports.

  • Test Cases: Test Case Risk Type, Test Case Risk Category, Test Case Risk Likelihood, Test Case Risk Impact, Test Case Risk Priority Number, Test Case Risk Extent of Testing.

  • Requirements: Requirements Risk Type, Requirements Risk Category, Requirements Risk Likelihood, Requirements Risk Impact, Requirements Risk Priority Number, Requirements Risk Extent of Testing.

Advance Query Reports

These fields are added under the Requirements and Test Cases tables to generate the advance query report: riskCategory, riskCategoryID, riskPriorityNumber, riskType, riskTypeID, impact, impactID, likelihood, likelihoodID, extentOfTesting, extentOfTestingID.

System Reports

The following two new reports have been added that include details related to Risk Based Analysis.