Test using TestNG

1 Introduction
2 Sample Test Result File
3 Reuse Manual Test Cases
4 Link Stories

Introduction

TestNG is a testing framework designed to simplify a broad range of testing needs, from unit testing (testing a class in isolation of the others) to integration testing (testing entire systems made of several classes, several packages, and even several external frameworks, such as application servers).

Writing a test is typically a three-step process:

Write the business logic of your test and insert TestNG annotations in your code.
Add the information about your test (e.g. the class name, the groups you wish to run, etc...) in a testng.xml file or in build.xml.
Run TestNG.

Here is how you can specify the Test Cases and Test Steps which will be created in QMetry as a part of the result files.

Automation Hierarchy	Test Cycle	Test Case	Test Step Tag

Automation Hierarchy

Test Cycle

Test Case

Test Step Tag

1 (default)

NA

Test Name is created as Test Case.

<test> tag under the <suite> tag will be considered as test case.
'name' attribute of <test> tag will be used as test case summary.

Test Method Name is created as Test Step.

<test-method> tag under <class> tag (with attribute "is-config" not set as true) will be considered as test case step.
'name' attribute of <test-method> tag will be used as test case step description.

2

Test Name is created as Test Cycle.

If there are multiple tests in the result file then multiple test cycles will be created based on the test name.

<test> tag under the <suite> tag will be considered as test cycle.
'name' attribute of <test> tag will be used as test cycle summary.

→ The parameter Test Cycle to Reuse will be ignored in the Automation.

Test Method Name is created as Test Case.

<test-method> tag under <class> tag (with attribute "is-config" not set as true) will be considered as test case case.
'name' attribute of <test-method> tag will be used as test case summary.

NA

3

Suite Name is created as Test Cycle.

If there are multiple suites in the result file then multiple test cycles will be created based on the suite name.

The <suite> tag will be considered as test cycle.
'name' attribute of <suite> tag will be used as test cycle summary.

→ If the value is provided for the Test Cycle to Reuse parameter, then the suite name in the files will be ignored. All the test case results for all suites and files will be uploaded in a single cycle similar to Hierarchy 1 (default).

Test Method Name is created as Test Cases.

<test-method> tag under <class> tag (with attribute "is-config" not set as true) will be considered as test case case.
'name' attribute of <test-method> tag will be used as test case summary.

NA

Note: If the Test Name or Test Suite Name length is more than 255 characters, the name will be truncated.

Supported Version: 1.2.5

Supported file types : XML

Sample Test Result File

Click to download the Sample Test Result File.

Automation Hierarchy 1

→ Test Case Key: The Test Case Key to be reused is mentioned in the Group. Methods, in which the Test Case Key will be reused, are placed inside the Group.

→ Story Key: Story Key is mentioned in the Group. Methods, which are to be linked to the story, are placed inside the Group.

Automation Hierarchy 2

Automation Hierarchy 3

Reuse Manual Test Cases

Keyword

Download Sample File

Description

testcasekey

While importing test result file, if you want to reuse test cases, then it is possible.

The mapping of test steps depends on the matchTestSteps parameter.

If the matchTestSteps parameter is set as “True” (Default): Create/Reuse a test case with a summary and test steps that exactly match the automated test case uploaded through the result file. The execution results and other execution details of the test case and steps will be imported from the automation result file.

If Test Case Summary and Test Step Summary (for all steps) match with the automated Test Case Name ( elements > name) and steps (elements > steps > name), Test Case Key and version will be reused.
- If Test Case Summary matches with the automated Test Case ( elements > name) but Test Step Summary does not match (for any of the steps), Test Case Key will be reused but a new version will be created.
- If Test Case Summary does not match, a new test case will be created.

If the matchTestSteps parameter is set as “False”:

When the Test Cycle is not been Reused

→ Create/Reuse a test case with a summary or test case key that exactly matches the automated test case uploaded through the result file, and exclude matching of test steps. The execution results of the test case will be imported or calculated based on the test case/step results from the automation result file. The execution result of the test case will be propagated to the test steps in the case of test case reuse/creation. Individual test case steps will not be matched and their execution results/details will not be picked from the result file.

When the Test Cycle is been Reused

When the Test Case Key is mapped in the result file and the Test Case Key is found linked to the Test Cycle.

The existing linked test case version, which is part of the Test Cycle will be used. If multiple versions of the same test case key are linked to the test cycle, the one which traced first will be used. The test steps will not be matched to create a new version or link a different version.

→ When the Test Case Key is mapped in the result file and the Test Case Key is not linked to the Test Cycle and the Test Case Key exists in the Test Case Library.

The existing latest version of the test case that matches based on the test case summary will be linked to the existing Test Cycle. If there are multiple test cases with the same summary exist, the one which is traced first will be linked to the existing Test Cycle. The test steps will not be matched to create a new version or link a different version.

→ When the Test Case Key is mapped in the result file and the Test Case Key is not found linked to the Test Cycle and the Test Case Key is not found in the Test Case Library, and the Test Case Summary matches with any existing test cases linked to the Test Cycle.

The existing linked Test Case version which is part of the Test Cycle will be used. The test steps will not be matched to create a new version.

→ The Test Case Key is not mentioned in the result file and the Test Case Summary matches any existing test case that is already linked to the Test Cycle.

The existing linked Test Case version which is part of the Test Cycle will be used based on the summary. The test steps will not be matched to create a new version.

→ When the Test Case Key is mapped in the result file and the Test Case Key does not exist in the Test Cycle OR is Not found in Library and the Test Case Summary also does not match any existing Test case in the Test Cycle and Test Case with the same summary is found in the Test Case Library.

The existing latest version of the test case that matches based on the test case summary will be linked to the existing Test Cycle. If there are multiple test cases with the same summary exist, the one which is traced first will be linked to the existing Test Cycle. The test steps will not be matched to create a new version or link a different version.

→ When the Test Case Key is mapped in the result file and the Test Case Key does not exist in the Test Cycle OR is Not found in Library and the Test Case Summary also does not match any existing Test case in the Test Cycle OR is not found in the Test Case Library.

A new test case without steps will be created and will be linked to the Test Cycle being reused.

The execution results of the test case will be imported or calculated based on the test case/step results from the automation result file. The execution result of the test case will be propagated to the test steps in the case of test case reuse/creation. Individual test case steps will not be matched and their execution results/details will not be picked from the result file.

In a project where propagation is off, the status of the step will not be mapped or changed.

Multiple test case keys (i.e. testcasekey) can be linked with a single test suite, which will update/create multiple test cases and link these test cases with an automated test suite. Multiple test case keys can be passed in the same tag e.g.

<group name="testcasekey:FIT-TC-1 FIT-TC-2 FIT-TC-3">

When Group(s) are defined at Suite level followed by a Test(s): If the Group's Class Name matches with the Test's Class Name, the Test Case Key/Story will be linked with those Tests.

Link Stories

Keyword

Download Sample File

Description

storykey

The specified stories will be linked to the created/reused test case.

When a valid story entity key is passed through the result xml file, the test case which is created using <class> is linked to the story.
When an invalid/archived story entity key is passed through result xml file, no story will be linked to the test case which is created using <class>.
Cross-project stories can also be linked to the test case.
Multiple storykey can be used in the same tag in the file to link multiple stories to the test cases. e.g.

<group name="storykey:FIT-1 FIT-2 OPT-3">

If multiple test case keys are mentioned, then the multiple stories will be linked with the test cases individually.
The users should have Test Case Edit and Requirement Edit permissions.
When Group(s) are defined at Suite level followed by a Test(s): If the Group's Class Name matches with the Test's Class Name, the Test Case Key/Story will be linked with those Tests.