Risk Based Testing: Auto Calculation of RPN and Extent Of Testing
Note: The Risk Based Testing feature is only available with the Advanced Features pack of QMetry.
Introduction
Risk-Based Testing (RBT) is a testing methodology that prioritizes software testing based on the probability of risk. This approach involves assessing risks by evaluating software complexity, business criticality, frequency of use, potential defect areas, and recent system changes. By targeting high-risk areas, RBT enhances testing effectiveness and efficiency.
QMetry allows users to define and measure risks, link them to requirements and test cases, and generate RBT reports.
Configuration
Required Permissions:
Users must possess the Project “View” rights in order to access and view the "Enable Risk Based Analysis" settings.
Users must possess the Project “Modify” rights in order to edit the "Enable Risk Based Analysis" Settings.
Steps
Go to Projects > Project/Release/Cycle.
Open the General Settings tab.
Move to Risk Analysis section on the screen.
If Enabled:
‘Risk Analysis’ fields are shown for unarchived test cases, requirements, Data Parameterized test cases, and Exploratory test cases.
If entities are copied within the same project, their values will be copied.
When entities are copied across projects, values will only be copied if RBT is enabled.
For Projects with Jira Integration the following fields are displayed on the Test Case detail page in Jira:
Risk Type
Risk Category
Likelihood
Impact
Risk Priority Number (RPN)
Extent of Testing
If Disabled:
The values are hidden but retained. If re-enabled, Risk Analysis values will be available to use.
Impact across QTM when RBT is enabled
Manage Fields
A) When RBT is enabled, four fields are auto created as System Fields within Customization > Manage Fields. Users have the flexibility to set them as mandatory, and to add, archive, delete, or customize their values as needed.
Risk Type
Risk Category
Likelihood
Impact
Lists
B) When RBT is enabled, the following lists are auto created under Customization > Lists section. Users can edit existing values and add new ones.
Risk Likelihood: The list is created with the values following values.
Value | Description | Value | Description |
|---|---|---|---|
1 | Least | 3.5 | Between High and Medium |
1.5 | Between Low and Least | 4 | High |
2 | Low | 4.5 | Between Very High and High |
2.5 | Between Medium and Low | 5 | Very High |
3 | Medium |
Risk Impact: The list is created with the following values.
Value | Description | Value | Description |
|---|---|---|---|
1 | Least | 3.5 | Between High and Medium |
1.5 | Between Low and Least | 4 | High |
2 | Low | 4.5 | Between Very High and High |
2.5 | Between Medium and Low | 5 | Very High |
3 | Medium |
Risk Type: The list is created with the values Project, Product, Process, and None.
Extent of Testing: The list is created with the following values:
E. T. Range (RPN Value) Accepted format >{Number}-{Number} | Value | E. T. Range (RPN Value) Accepted format >{Number}-{Number} | Value |
|---|---|---|---|
>0 - 5 | Report bugs only | > 20 - 25 | Extensive |
> 5 - 10 | Opportunity | Out of Range | Not defined |
>10 - 15 | Cursory | ||
> 15 - 20 | Broad |
Risk Category: The list is created with the following values.
Risk Category | Description |
Competitive Inferiority | Failures to match competing systems in quality. |
Data Quality | Failures in processing, storing, or retrieving data. |
Date and Time Handling | Failures in date-based and/or time-based inputs/outputs, calculations, and event handling. |
Disaster Handling and Recovery | Failure to degrade gracefully in the face of catastrophic incidents and/or failure to recover properly from such incidents. |
Error Handling and Recovery | Failures due to bad inputs, beyond peak, or other illegal conditions (i.e., knock-on effects of deliberately inflicted errors). |
Functionality | Failures that cause specific features not to work. |
Installation, Setup, Upgrade, and Migration | Failures that prevent or impede deploying the system, and migrating data to new versions, including unwanted side-effects (e.g., installing additional, unwelcome, unintended software such as spyware, malware, etc.). |
Interoperability | Failures occur when major components, subsystems, or related systems interact. |
Load, Capacity, and Volume | Failures in scaling of system to expected peak concurrent usage levels. |
Localization | Failures in specific localities, including languages, messages, taxes and finances, operational issues, and time zones |
Networked and Distributed | Failure to handle networked/distributed operation, including latency, delays, lost packet/connectivity, and unavailable resources. |
Operations and Maintenance | Failures that endanger continuing operation, including backup/restore processes. |
Packaging/Fulfillment | Failures associated with the packaging and/or delivery of the system or product. |
Performance | Failures to perform as required under expected loads. |
Portability, Configuration, and Compatibility | Failures specific to different supported platforms, supported configurations, configuration problems, and/or cohabitation with other software/systems. |
Reliability, Availability, and Stability | Failures to meet reasonable expectations of availability and mean-time-between-failure. |
Security/Privacy | Failures to protect the system and secure data from fraudulent or malicious misuse. |
Standards Compliance | Failure to conform to mandatory standards, company standards, and/or applicable voluntary standards. |
States and Transactions | Failure to properly respond to sequences of events or particular transactions. |
Usability | Failures arise from aspects of the system that make players or other users feel ineffective, inefficient, or dissatisfied while using the system. |
User Interface | Failures where incorrect information is presented directly to users. |
Requirements Module
The Risk Analysis section appears on the create and edit screens for requirements. It includes:
Risk Type: Select from Project, Product, Process, or None.
Risk Category: Categorize the risk.
Likelihood: Select from 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5 (1 being the least and 5 being very high).
Impact: Select from 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5. (1 being the least and 5 being very high).
Risk Priority Number (RPN): Auto-calculated as RPN = Likelihood x Impact.
Extent of Testing (ET): Auto-calculated based on RPN value.
Test Cases Module
The Risk Analysis section on the test case creation and edit screens includes the following fields:
Risk Type: Select from Project, Product, Process, or None.
Risk Category: Categorize the risk.
Likelihood: Select from 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5 (1 being the least and 5 being very high).
Impact: Select from 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5. (1 being the least and 5 being very high).
Risk Priority Number (RPN): Auto-calculated as RPN = Likelihood x Impact.
Extent of Testing (ET): Auto-calculated based on RPN value.
Test Case Created from Requirement
When a test case is created from a requirement, ‘Risk Analysis’ fields are auto populated from the requirement.
Tabs/Screens on which RBT Fields will be visible
Risk Analysis columns are displayed on the following screens. The Filters section includes fields to filter records using RBT fields.
Test Case > Requirements tab
Test Case > Requirement tab > Link Requirements
Test Case Bulk Operations
Requirements > Test Cases tab
Requirement Bulk Operations
Test Suite > Test Cases tab
Issues > Requirements tab
Issues > Requirement tab > Link Requirements
Layout Management
Fields related to Risk Analysis are available in the Requirements and Test Cases modules under Layout Management section of Customization. Users cannot move or hide these fields, they must be placed under the risk analysis section only.
Requirements and Test Case Import and Export
Import: All Risk Analysis field values are included when importing requirements and test cases.
Export:
For XLSX and CSV exports, all Risk Based Testing field values are exported along with other details.
For PDF and Word exports, the Risk Based Testing section is included.
You can bulk export test case details with all Risk Analysis fields from the Test Suite > Test Cases tab.
Visual Reports
Users can generate tables and Add gadgets using Risk Fields.
The fields related to Risk Based Testing are available for requirements and test cases to generate the reports.
Test Cases: Test Case Risk Type, Test Case Risk Category, Test Case Risk Likelihood, Test Case Risk Impact, Test Case Risk Priority Number, Test Case Risk Extent of Testing.
Requirements: Requirements Risk Type, Requirements Risk Category, Requirements Risk Likelihood, Requirements Risk Impact, Requirements Risk Priority Number, Requirements Risk Extent of Testing.
Advance Query Reports
Users can generate Requirement and Test Case Reports that include the following fields: risk type, risk category, risk impact, risk likelihood, risk priority, and the estimated extent of testing with the following SQL query.
select testcases.EntityKey, testcases.Risktype, testcases.RiskCategory,testcases.Impact, testcases.Likelihood, testcases.RiskPriorityNumber, testcases.ExtentOfTesting FROM testcases WHERE testcases.projectName = 'Fit Tracker' Group By testcases.riskCategory
System Reports
The following two new reports have been added that include details related to Risk Based Analysis.
Risk Traceability for Requirements: Refer to https://qmetrysupport.atlassian.net/wiki/spaces/QTMSv8230/pages/4711024608 for more details.
Risk Traceability for Test Cases: Refer to https://qmetrysupport.atlassian.net/wiki/spaces/QTMSv8230/pages/4711024706 for more details.