Integrating TestRail with Nightwatch.js

February 7, 2021

#software testing

18

#quality assurance

17

#nightwatchjs

15

#post'

4

#testrail

1

Making Test Case Management Systems Useful

If your tests suites are manual and scattered about in various spreadsheets, Google Docs, DropBox, notepad and the like then organizing them into a centralized Test Case Management System makes a lot of sense. Without the organization they afford you the tests tend to get lost to time and personnel changes. In addition, tracking test results and when they were run last is dicey. As a manager, reporting on quality is nerve-wracking.

However, what if all your tests are automated into a continuous testing pipeline?

What benefit does a Test Case Management System offer for automated tests?

To an automated test engineer, not a whole lot. Most continuous testing frameworks run the tests constantly and retain their pass/fail history already. Historically, the test case management systems that were implemented required a tester to manually update the test results in the TCMS after the automation ran which is a soul-crushing waste of time when your automation framework has it already. I've been critical of and fought, hard, against them based on prior bad experiences.

Most systems have a mix of both manual and automated tests, though.

Rarely do most software companies have every single test automated. This means there will still be manual tests you need to keep track of and report on. The test case management system provides a central place to store the details of both the automated and manual tests along with their run data.

Besides organization and centralized reporting, the Test Case Management System helps management present on test coverage. For non-technical users, it helps them see what a test is doing without looking in code. It also helps with auditing.

Wouldn't it be great if we could get the utility of a Test Case Management System automatically without extra, especially manual, work for the test engineers?

We recently did just that with TestRail and NightwatchJS on my latest test project.

Integrating TestRail with Nightwatch.js

My team and I were recently tasked with a project to write test automation to verify demo environment functionality prior to use. Previously, these tests were performed manually using a checklist in a Word document. It seemed like a good opportunity to move those tests into TestRail and then automate them using Nightwatch.

However, if I automated the tests I didn't want to deal with manually updating the test case management system, TestRail.

Fortunately, TestRail provides an API that we leveraged to send the test results from Nightwatch back to the test run in TestRail.

Using the combination of the two, our business users could see the test results, we didn't have to do any extra work (beyond the initial setup), and all the tests were centrally organized.

How to Update TestRail with Nightwatch Test Results Automatically

I've published an NPM package called nightwatch-testrail-updater that you can import into your Nightwatch test suite. It passes the results of the test case through the TestRail API when the test finishes executing.

Outside of the overall pass status, the individual test steps are logged to the test case results and comments section as shown below.

Installing nightwatch-testrail-updater

To add nightwatch-testrail-updater to your project, from a command line in your test suite, run

npm install nightwatch-testrail-updater --save

After, in your Nightwatch configuration file, nightwatch.json, add or append

"custom_commands_path": ["./node_modules/nightwatch-testrail-updater/commands"]

This will make nightwatch-testrail-updater's updateTestRail() method available as a custom command in the Nightwatch browser object.

Also in nightwatch.json's test_settings section add a section called testRail with the host of your TestRail deployment and the runId of the TestRail suite containing your tests. The runId is the "R" number shown in the upper left of the test run in TestRail when you are viewing the test run details.

"test_settings": {
    "default": {
      "globals": {
            "testRail": {
                 "host": "testrail.mycorp.com",
                  "runId": "12345"
            }
        }
    }
}

The TestRail API will need your TestRail username and API Key. Create environment variables on the system named TESTRAIL_USERNAME and TESTRAIL_API_KEY to store your TestRail username and API key information in order to authenticate with the TestRail API (example setting environment variables.

Sending Nightwatch Test Results to TestRail using nightwatch-testrail-updater

In order to associate your automated tests to the entries in the TestRail test run you need to add the TestRail test ID in your test.

So, in your test, add a line called browser.testId = testIdHere

Set the value equal to the numeric test case ID, the number prefixed by "C" when viewing the list of test cases in the TestRail Test Cases tab. This value is used in the API call to update the test result in TestRail.

'descriptive test case name': function (browser) {
        // For test case id C112233
        browser.testId = 112233;

        // Test code here
    }

In the afterEach test hook in your Nightwatch test suite add a call to .updateTestRail() so that it runs after each test finishes executing

module.exports = {
...
    afterEach: function (browser, done) {
        browser.updateTestRail(browser);
        browser.end();
    }
...
}

After each test case in the suite finishes the TestRail API will have the result posted to it. See also the GitHub readme for nightwatch-testrail-updater or my TestRail Nightwatch example project.

Final Thoughts

Test Case Management System features typically benefit managers and business users to a higher degree than the engineers and test authors who are closer to the individual tests. To win over test engineers, the ones who will be tasked with getting the data into the TCMS, ensure your TCMS can integrate seamlessly with their test automation. Manual entry would be seen as redundant work. Test Case Management Systems that provide an API like TestRail paired with an extensible test framework like Nightwatch allow for that automatic integration to allow for successful adoption across your organization.

Let me know what you think and if you end up using this where you work! Reach out through my social links below 👇