QMetry Intelligence - Frequently Asked Questions

General Questions

What are the cost implications for using the QMetry Intelligence AI features?

• Free: The “Auto Generation of Test Case and Steps using QI” and “Flaky Score” and “Success Rate” calculation features are free for a limited time. These features may become paid in the future.

• Credit Balance: “AI-powered Virtual Search Assistant” requires an Open AI API Key from an active Open AI account with a sufficient credit balance. Uploading the documents and performing a refresh of the documentation uses tokens deducted from the “gpt-4o-mini" credit balance.

Are there any set limits for using all QMetry Intelligence AI Features?

• No limits have been imposed for Auto Generation of Test Case and Steps at the moment.

• Flaky score can be created once in 24 hours,

• AI-powered Virtual Search Assistant limit is based on the credit balance for the OpenAI keys configured for this feature.

What training and awareness roadmap is in place for AI technology users? How is citizen readiness assessed to equip users with the necessary skills?

AI Introductory Demo and Documentation are available to help user readiness in using the feature. Please write to sales@qmetry.com with your details to schedule a training or request a demo. Documentation links are available at the end of this page.

QMetry Intelligence Features fall into which categories?

• Auto Generate Test Cases and Steps using QI feature - Self Built Gen AI.

• AI-Powered Virtual Search Assistant feature - Public Gen AI.

• Flaky Score and Success Rate features - Not Gen AI. ML algorithms on inhouse built model.

Does QMetry provide an option to disable the QMetry Intelligence AI capabilities within the application and/or restrict access to certain users if needed to? 

Yes, the QMetry Intelligence features can be enabled/disabled on the instance level by QMetry instance admin anytime through Customization > General Settings & Audit > AI Configurations section.

All the QMetry Intelligence features will be disabled by default.

What are the benefits of enabling QMetry Intelligence within a test management solution? 

QMetry Intelligence is a sophisticated suite of features that leverages Artificial Intelligence and Machine Learning advancements to accelerate testing cycles and enhance test management efficiency. The key benefits of the AI features include streamlining your testing process, enhancing test case creation efficiency, and improving the overall test result reliability. QI features directly help reduce your dollar value and will enhance your overall spending and saving.

Are QMetry Intelligence capabilities included in the base package or the advanced package?

The QI capability is included with both the packages and instance administrators have the option to enable/disable the AI feature at the instance level.

Can the AI models for QMetry Intelligence features be fine-tuned on specific requirements or specific use-cases?

QMetry Intelligence features utilize pre-trained AI models and hence customer data is not required or utilized to train these models for any specific use-cases. Customizations are not applicable for QMetry Intelligence AI use cases.

Do the QI features use the same AI models or different models for each feature?

• Auto Generate Test Cases and Steps using QI features utilizes Meta AI’s pretrained LlaMA2 model.

• AI-Powered Virtual Search Assistant using QI utilizes Open AI gpt-4o-mini.

• Flaky Score and Success Rate features use ML algorithms on internally built model.

Can we control which project/role can access QMetry Intelligence features?

QMetry Intelligence features can be enabled/disabled at the instance level only.

What is the percentage% time/effort a team can save by using the AI Features?:

Estimated 30-40% time savings have been observed by utilization of QMetry Intelligence features.

Privacy and Security

How does QMetry ensure the privacy and security of the data? 

QMetry validates every AI request from the UI and backend before forwarding it to the AI models for processing. The models utilized are pretrained, and QMetry does not employ any customer data for training purposes. Additionally, AI services are hosted on AWS servers on a private subnet and are not publicly accessible.  Queries submitted to QI features are treated as “Customer Content“ according to our terms of use.

Will my data be used to train the QMetry Intelligence AI Models?

No. QMetry uses pre-trained models for the feature to “Auto Generate Test Cases and Steps using QI”.

What data QMetry will access for the QMetry Intelligence features within the QMetry Test Management Enterprise? 

The AI models used across the QMetry product are stateless and pre-trained. QMetry does not utilize any customer data to train our AI models. Additionally, no customer data provided entered in the QMetry Intelligence features is retained.

Does any data used for QMetry Intelligence features, stored in QMetry or sent to other AI service provider? (Information on Data Security and Compliance)

• For “Auto Generate Test Cases and Steps using QI”, QMetry does not store or retain any data.

• For “AI Powered Virtual Search Assistant“, if a user searches for questions related to user-provided documents, the query and part of the document is sent to the Open AI Services, but the communication is stateless. In this case, the privacy and security policies of Open AI are applicable.

Is my data shared with any third-party AI Service Providers?

• For “Auto Generate Test Cases and Steps using QI”, QMetry does not store or retain any data.

• For “AI Powered Virtual Search Assistant“, if a user searches for questions related to user-provided documents, the query and part of the document is sent to the Open AI Services, but the communication is stateless. In this case, the privacy and security policies of Open AI are applicable.

Does QMetry retain any of the user data provided for the QMetry Intelligence features? 

QMetry does not retain any of the customer data.

Does my QMetry instance-related data submitted for QMetry Intelligence features be visible on other instances in any manner?

QMetry customer instances are logically separated, and no data including those entered for QMetry Intelligence features would be visible or accessible on other instance.

How does QMetry ensure that AI software and frameworks stay up to date with the latest security patches and versions? What is the process for regular reviews and updates of libraries, dependencies, and operating systems?

QMetry does not directly upgrade to the next version or switch models for QMetry Intelligence. Our AI labs continuously monitor industry innovations and validate new models and upgraded versions. When changes are necessary, QMetry's AI labs conduct a Proof of Concept (POC). This process entails research and development, along with a feasibility study of the new model, to ensure it precisely meets the use cases, maintains accuracy, and is cost-effective. This approach guarantees that we verify the superiority of the output from the upgraded or newer models. Additionally, we ensure that the new models generate data that meets or exceeds the standards set for backward compatibility.

What are the mitigating controls in place to prevent data exfiltration due to AI capabilities?

• Feature “AI Powered Virtual Search Assistant“ – QMetry AI servers save only the customer documents on the server. This is protected under private subnets. The data is transformed, encoded and encrypted hence can’t be retrieved in its original form. All customer data have their separate embedding space which prevents cross customer sharing of data. Authorization and authentication mechanisms are in place to control access of data.

• Feature “Auto Generate Test Cases and Steps using QI” – The APIs are stateless. QMetry AI servers do not store any data. The model is pretrained and customer data is not used for training.

• Feature “Flaky Score and Success Rate” – No data is stored. The APIs are stateless.

Is the Large Language Model (LLM) isolated, and is Zero Trust enforced in the QMetry Intelligence AI implementations?

Yes

How are the Principle of Least Privilege and Identity and Access Management enforced? What strict access controls are in place to ensure only authorized personnel can access sensitive data?

Only a few authorized personnel from QMetry have the access to the servers. No data is stored on the AI servers.

Is a sandbox environment available for isolation review and testing of the AI features? Will regular security assessments be performed against the AI system to identify vulnerabilities?

Yes to both.

How is sensitive or proprietary data handled to ensure compliance with data classification policies?

Unless the user uploads the data in QMetry that is sensitive or proprietary. Any data is stored just on QMetry Servers and does not leave QMetry infrastructure.

What monitoring controls are in place for the AI system? How do you ensure system integrity and security?

Customers cannot monitor or control the QMetry Intelligence capabilities. QMetry uses pre-trained models, and the maintenance of these models is performed by QMetry.

AI-Powered Virtual Search Assistant

Are there any limits for utilizing the AI Search Assistant?

“AI-powered Virtual Search Assistant” requires an Open AI API Key from an active Open AI account with a sufficient credit balance. Uploading the documents and performing a refresh of the documentation uses tokens deducted from the “gpt-4o-mini" credit balance.

Is the Open API key token managed at user level or instance level?

The API key from Open AI is configured at an instance level in QMetry.

Do we need only one OpenAI access token per QMetry instance? What are the best practices for creating and managing this access token?

Yes, requires an Open AI API Key from an active Open AI account with a sufficient credit balance.  Uploading the documents and performing a refresh of the documentation uses tokens, deducted from the “gpt-4o-mini" credit balance. QMetry does not manage Open AI keys or mapping of instances and keys. Hence one key can be used for multiple QMetry instances.

The AI feature then uses this key to interact with OpenAI which is charged to their OpenAI account. OpenAI charges are based on tokens. The number of tokens used depends on the amount of documentation uploaded to the AI system and the type of search queries and their results. This is directly controlled by OpenAI.  Use of this feature is optional and can be disabled in the system on the configuration page.

If a document is uploaded to QMetry Intelligence for Virtual Search Assistant, will it be shared with OpenAI or made publicly available?

When a user searches for questions related to user-provided documents, the query and part of the document is sent to the Open AI Services, but the communication is stateless. In this case, the privacy and security policies of Open AI are applicable. Entire document is not uploaded.

Can the Virtual Search Assistant feature interpret Images or tables provided in a document/URL?

This feature is currently unsupported.

Auto Generate Test Cases and Steps using QI

How is the accuracy of data ensured?

QMetry utilizes pre-trained Meta AI's LlaMA2 models to automatically generate test cases based on use cases. These models require input from a user description which should include a use case with relevant background context and acceptance criteria. The accuracy of the output is contingent upon the quality and content of the input.

Best Practices for Generating Accurate Test cases:

To ensure QI generates test cases with higher accuracy, the requirement description should adhere to the following guidelines:

• Structured Use Case Format: The requirement description must be presented in a well-structured use case format to facilitate accurate test case generation.

• Acceptance Criteria: Include clear acceptance criteria that define the expected behavior of the feature. Decomposing stories into detailed criteria will improve the accuracy of the generated test cases.

• Contextual Analysis: Providing these details allows QI to analyze the context effectively and accurately identify relevant test scenarios.

• Description Length: Descriptions should be between 30 and 1500 words.

• Description Details: Description shall provide sufficient detail for QI to analyze requirements, identify critical scenarios, and generate accurate test cases.

Is there any standard template that users need to input in the description for the AI model to generate good quality test cases? 

There is no mandatory template needed for auto test case generation. However, it is recommended to provide user description in a manner that includes use case, acceptance criteria, implementation approaches in as much detail possible to generate high quality test cases. The additional context helps AI models to understand the requirement while generating the test cases.

How should a user authenticate for the Auto Generation of Test Cases feature using Meta Llama2? What steps are required for token creation or other authentication methods?

This feature needs to be enabled by the QMetry instance administrator and would be ready to use out of the box. 

Is the Auto Generation of Test Cases Feature capable of creating scenarios in different languages?

This is currently supported only in English language.

Can the AI Model generate tests based on the base document/guidelines that users provide?

At the moment, there is no possibility to create tests around specific user-defined documents.

Can the Auto Generation of Test Cases feature generate Automation Test Cases/ BDD Scenarios

Currently Functional and Manual Test Cases are supported. Automation Test Cases or BDD scenarios are not supported.

Contact and Support

How do I contact customer support?

QMetry Support can be reached by signing up to the Support Portal using your email address and submitting a ticket. Alternatively send an email to qtmprofessional@qmetrysupport.atlassian.net.

How do I access Help Documentation for QMetry Intelligence Features?

QMetry Intelligence Help Documentation