Sauce Labs Reporting & Selenium Test Results

Sauce Labs is a popular service that allows you to run automated Selenium tests (as well as many other frameworks) against various browser, OS and version combinations. When you develop your frontend tests and want to run them against many different platforms, it can be time-consuming and difficult to set up and maintain a large Selenium Grid installation with all these configurations.

Selenium cloud services such as Sauce Labs provide a ready-to-use and scalable platform to run your browser tests quickly. In this guide we will look at all the details to get started:

Setting up the initial project
Starting with Selenium automation
Running our tests against Sauce Labs
Setting up a full test suite with Sauce Labs
Reporting results to test management
Integrating with GitHub Actions CI

If you are new to Selenium, you might also find our complete guide on Selenium test automation useful. Let's look at Sauce Labs reporting and test automation next!

Reporting Selenium and Sauce Labs test automation runs to Testmo

Initial Project & Repository Setup

For this article and example project we are setting up a new GitHub repository called example-saucelabs-reporting. You can also use a different service such as GitLab or Bitbucket if you prefer (we will be using GitHub Actions for our CI pipeline in this example, but other CI tools will also work). You can find the full project repository on GitHub so you can always review the full files there.

We will integrate our tests with GitHub Actions later in this article to run our tests on GitHub's CI platform. But it's still useful to have a way to run our automation scripts, test suite and dev tools locally so we can try everything before committing changes to our repository. We could run everything directly on our machine, but using Docker makes things much easier (and more secure).

We will use a simple Docker Compose config for this project so we can set up and use a local development environment:

# dev/docker-compose.yml
version: '3'
services:
  node:
    image: node:19
    volumes:
      - ./../:/project
    working_dir: /project

This is the same approach we used in our earlier article on Selenium, so you can also read about all the details there. We will use the official node (Node.js JavaScript runtime) container image, as our example project will use JavaScript to run our Selenium browser automation. To start the container and launch a shell inside it, we use the docker compose run command:

$ docker compose run node bash
Creating dev_node_run ... done
root@d433d79213d7:/project$ # in the container

First Sauce Labs Browser Automation

Before running our first browser automation script against Sauce Labs, let's look at the script details here. We will create a new file named automate.mjs in our project directory with the following content (note the file extension of .mjs, which enables the newer ES6 JavaScript features).

As noted before, we will use JavaScript for our example code in this article. We will also use the official selenium-webdriver JavaScript package as our Selenium client library. However, it's useful to note that you can use any programming language and any Selenium or WebDriver library to follow this example and run your browser automation with Sauce Labs. We just choose JavaScript and selenium-webdriver here because it's very easy to get started. But you can also use any other platform and still follow this guide, such as:

Node.js/JavaScript with WebdriverIO
Python with the selenium package
C# with the Selenium library
Java with the Selenium library
PHP with Facebook's php-webdriver library
Ruby with selenium-webdriver

// automate.mjs
import { Builder, By, Key, until } from 'selenium-webdriver';

if (!process.env.SAUCE_USERNAME || !process.env.SAUCE_ACCESS_KEY) {
    console.log('Sauce Labs user name or access key not set.')
    process.exit(1);
}

// We connect to Sauce Labs's Selenium service
const server = process.env.SAUCE_URL || 
	'https://ondemand.us-west-1.saucelabs.com/wd/hub';

// Sauce Labs authentication and options
const sauceOptions = {
    username: process.env.SAUCE_USERNAME,
    accessKey: process.env.SAUCE_ACCESS_KEY,
    name: 'First browser automation'
};

let browser = process.env.BROWSER || 'chrome';
// Microsoft uses a longer name for Edge
if (browser == 'edge') {
    browser = 'MicrosoftEdge';
}

// Set up a new browser session
let driver = await new Builder()
    .usingServer(server)
    .forBrowser(browser)
    .setCapability('sauce:options', sauceOptions)
    .build();

try {
    // Automate DuckDuckGo search
    await driver.get('https://duckduckgo.com/');

    // Search for 'Selenium dev'
    const searchBox = await driver.findElement(By.id('search_form_input_homepage'));
    await searchBox.sendKeys('Selenium dev', Key.ENTER);

    // Wait until the result page is loaded
    await driver.wait(until.elementLocated(By.css('#links .result')));
} finally {
    // Close the browser
    await driver.quit();
}

This is our first automation script that we will use to try Sauce Labs and drive our first browser session. Let's review our script step by step:

Sauce Labs config (lines 9-18): We need to configure a couple of settings first. Namely we set up the Sauce Labs Selenium service URL as well as the username, access key and session name. The script expects environment variables named SAUCE_USERNAME and SAUCE_ACCESS_KEY to be available (you get the values from your Sauce Labs account).
Browser name (lines 20-24): Sauce Labs makes it easy to run our script against different browsers (see below). So our script allows us to set an environment variable called BROWSER with values such as chrome, firefox, edge etc. If no variable is set, we just default to chrome.
Starting Selenium session (lines 26-31): We build our Selenium driver object and pass all required settings, such as the Sauce Labs address, browser name, authentication values etc. This also starts the browser session and launches the browser window on Sauce Labs.
Browser automation (lines 34-42): We can then use the Selenium webdriver API to automate the browser. In our example we access the DuckDuckGo search engine, search for Selenium dev and wait for the result page to be loaded. Make sure to review our Selenium test automation article to learn more about using the Selenium API.
Closing browser (line 45): It's important to close the browser window and end the Selenium session, so Sauce Labs can mark the session as completed. We run quit inside a try/finally block to ensure that the browser is always closed, even if there was an error in our script.

Running Our Code Against Sauce Labs

Let's run our code and automate our first browser session with Sauce Labs. Inside our container (see above) we install the required JavaScript package. We are using the official selenium-webdriver package:

# Run this inside the container
$ npm install --save-dev selenium-webdriver

This will also create our package.json and package-lock.json files in our project directory. These files will make it easy to install all our project dependencies on other machines, e.g. during our CI pipeline run. Make sure to commit these files to the repository as well.

We can now use the node command line tool to run our script. But first we need to set the required environment variables, namely SAUCE_USERNAME and SAUCE_ACCESS_KEY. As mentioned, you can find these details in your Sauce Labs account.

Once the environment variables have been set, we can run our script for the first time. We can also override the default browser by setting an environment variable. This way we can select different browsers to run our automation script with Sauce Labs:

# Run all these commands inside the container

# Start by setting the required environment variables
$ export SAUCE_USERNAME=*************
$ export SAUCE_ACCESS_KEY=*************

# Running our script against Sauce Labs with different browsers
$ BROWSER=chrome node automate.mjs
$ BROWSER=firefox node automate.mjs
$ BROWSER=edge node automate.mjs

If everything works correctly, our script connects to Sauce Labs, starts a browser session, runs our DuckDuckGo search and finally ends the session. You can view all steps and a full video of the browser session with all changes & browser interactions in Sauce Labs:

A single browser automation session in Sauce Labs

Complete Browser Test Automation Suite

So far we have used our script to automate a browser session to access a search engine. But we haven't really tested anything yet. To actually write tests with Sauce Labs, we need to use a testing framework and then add assertions (checks) to verify that the web page we test behaves the way we expect.

In our concrete example, we will not just access the search engine and submit a search query. We will also check that the result page contains a specific web address. If the result contains the web address we are looking for, we mark the test as passed. If the web address is missing, we fail the test. Likewise, if you test more complex web pages and forms, you would add various checks to the test to ensure the page works as expected.

In our example we will use a simple JavaScript testing framework called Mocha/Chai. We install the framework with NPM again, which also adds the new packages to our package.json file:

# Run this inside your container
$ npm install --save-dev mocha chai mocha-junit-reporter

We add a new file named test.mjs to store our test suite. Remember that you can always find the full repository files on GitHub. Let's review some of the details of this script:

Test setup: Our beforeEach function is called by the testing framework before each test is executed. Here we are setting up our Sauce Labs connection again and start a new browser session. We will use the same environment variables for the authentication and browser name. This time we will also pass the actual test name to Sauce Labs, so we can see our test names for the browser sessions. It's a good idea to start a new browser session for each test like we do, so we always start with a fresh browser without any previous cookies, page history or cached files.
Test teardown: After each test run, the framework will call our afterEach function. We stop the browser session here and close the browser. Before that we are also sending the result of our test to Sauce Labs (passed or failed). This is not strictly required, but it's nice to be able to see the test status in Sauce Labs as well. We use their special syntax to send the result via the executeScript function.
Test cases: At the end of our test suite file we have our various test cases. We also define a helper function (search) that implements the actual search and returns the result page content. Our test cases call this function to search DuckDuckGo with a specific search term and then check for a web address in the search result. The assert.isTrue call throws an error if the passed check failed, so we can signal to our testing framework if our check passed or failed.

// test.mjs
import { Builder, By, Key, until } from 'selenium-webdriver';
import { assert } from 'chai';

describe('search engine', async function () {
    let driver;

	// [..]

    // Before each test, initialize Selenium and launch browser
    beforeEach(async function() {
        // We connect to Sauce Labs's Selenium service
        const server = process.env.SAUCE_URL ||
            'https://ondemand.us-west-1.saucelabs.com/wd/hub';

        // Sauce Labs authentication and options
        const sauceOptions = {
            username: process.env.SAUCE_USERNAME,
            accessKey: process.env.SAUCE_ACCESS_KEY,
            name: this.currentTest.fullTitle()
        };

        let browser = process.env.BROWSER || 'chrome';
        // Microsoft uses a longer name for Edge
        if (browser == 'edge') {
            browser = 'MicrosoftEdge';
        }

        driver = await new Builder()
            .usingServer(server)
            .forBrowser(browser)
            .setCapability('sauce:options', sauceOptions)
            .build();
    });

    // After each test, submit the result and close the browser
    afterEach(async function () {
        if (driver) {
            // Send test result to Sauce Labs
            const result = this.currentTest.state == 'passed' ?
                'passed' : 'failed';
            await driver.executeScript(`sauce:job-result=${result}`);

            // Close the browser & end session
            await driver.quit();
        }
    });

    // A helper function to start a web search
    const search = async (term) => {
        // Automate DuckDuckGo search
        await driver.get('https://duckduckgo.com/');
        const searchBox = await driver.findElement(
            By.id('search_form_input_homepage'));
        await searchBox.sendKeys(term, Key.ENTER);

        // Wait until the result page is loaded
        await driver.wait(until.elementLocated(By.css('#links .result')));

        // Return page content
        const body = await driver.findElement(By.tagName('body'));
        return await body.getText();
    };

    // Our test definitions
    it('should search for "Selenium"', async function () {
        const content = await search('Selenium');
        assert.isTrue(content.includes('www.selenium.dev'));
    });

	// [..]
});

In our package.json file we include a couple of script aliases to make it easier to run our test suite for this example project. You can see the package config file in the repository. We can just use npm run to call one of our script aliases. So inside our container, simply run the tests like this (remember that the above mentioned Sauce Labs authentication variables still need to be set in case you restart the container):

# Running our Selenium test suite against Sauce Labs
$ npm run test     

> test
> npx mocha test.mjs

  search engine
    ✔ should search for "Selenium" (7382ms)
    ✔ should search for "Appium" (6253ms)
    ✔ should search for "Mozilla" (6286ms)
    ✔ should search for "GitHub" (6246ms)
    ✔ should search for "GitLab" (6952ms)

  5 passing (1m)

After the test run completed, our testing framework prints the test results to the console. You can also see the various browser sessions in Sauce Labs. Because we included the test names for our browser session names, it's easy to identify the sessions:

Our Selenium test sessions including result statuses in Sauce Labs

Reporting Results to Test Management

We will now look at Sauce Labs reporting of test results and will send the test runs to our test management tool Testmo. Reporting the test runs makes it easy to track our results, share tests with the entire team, identify slow or flaky tests, improve test performance and compare test runs.

We will use the testmo command line tool to submit our results, so we will install the required package first (note that you can still use this tool even if you don't use JavaScript/Node.js for your project otherwise; NPM makes it easy to install and deploy tools for projects of any language/platform):

# Run this inside your container
$ npm install --save-dev @testmo/testmo-cli

In addition to our Sauce Labs environment variables, we will also need to configure additional variables for Testmo. We first set the Testmo web address of our account and then set an API key for our user (which can be generated from the user profile page in Testmo).

The testmo tool can submit a test run with the automation:run:submit command. We just specify a few additional details such as the project ID, new test run name, configuration etc. In our example we specify the browser name as a configuration (Chrome, Firefox etc.). Make sure to add these configurations in Testmo under Admin > Configuratons. Or you can simply remove the --config option from the command.

Note that we pass npm run test-junit at the end. By doing this, the testmo tool calls this command. Doing this (instead of calling this command separately before) has the advantage that the Testmo command line tool can also capture the full console output, exit code and testing times automatically. And the test-junit script alias not just runs our tests, it also writes the test results to a JUnit XML file, which is the de facto standard file format to exchange test results between tools (testmo also expects this format).

Alternatively we can also just run npm run test-ci here, which is a script alias defined in package.json that runs the entire above testmo command line.

# First set the required variables inside the container
$ export BROWSER=chrome
$ export TESTMO_URL=*************
$ export TESTMO_TOKEN=*************

# Then execute tests and report results to Testmo
$ npx testmo automation:run:submit \
	--instance "$TESTMO_URL" \
	--project-id 1 \
	--name "Sauce Labs test run for $BROWSER"
	--config "$BROWSER" 
	--source "frontend" 
	--results results/*.xml 
	-- npm run test-junit # Note space after --

# Or we can run the same command with our script alias
$ npm run test-ci

After we use the command line to submit and report our Sauce Labs test run, we can see all the tests, results, test times, failures etc. in Testmo. This provides a great overview of our results and also makes it easy to report issues (such as failing tests) to a linked issue tracker such as Atlassian Jira. This is what a test run and its results look like in Testmo:

View a Sauce Labs test automation run with all results in Testmo

CI Pipeline Integration (GitHub Actions)

We have now developed a test automation suite to run and verify web searches with live browsers running on Sauce Labs. But wouldn't it be nice if we could run these tests automatically as part of a CI pipeline? This is exactly what we will set up next.

For our example we will use GitHub Actions (because our repository is hosted on GitHub). But you can use the same approach with any other CI/CD service. For example, we have additional articles on setting up and using test automation suites with various other services to help you get started:

For our GitHub Actions CI pipeline we create a new workflow config and store it in our project under .github/workflows/test.yml. Here's the complete workflow:

# .github/workflows/test.yml
name: Test

on: [push, workflow_dispatch]

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest

    container:
      image: node:19

    strategy:
      fail-fast: false
      matrix:
        browser: ['chrome', 'firefox', 'edge']

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: '19'
          cache: 'npm'
      - run: npm ci
      - run: npm run test-ci
        env:
          BROWSER: ${{ matrix.browser }}
          SAUCE_USERNAME: ${{ secrets.SAUCE_USERNAME }}
          SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }}
          TESTMO_URL: ${{ secrets.TESTMO_URL }}
          TESTMO_TOKEN: ${{ secrets.TESTMO_TOKEN }}

This workflow config launches three separate test steps, one for each browser, namely for Google Chrome, Mozilla Firefox and Microsoft Edge. So all our tests will automatically be tested against these browsers. GitHub Actions will also run our tests in parallel (because we defined a matrix for parallel testing here). Depending on the Sauce Labs account subscription you have, multiple browsers would run in parallel to speed up test execution. You can also learn more about parallel Selenium testing in our separate article.

Note the various environment variables we set at the end of our workflow configuration with the Testmo and Sauce Labs settings. These are defined as Secrets in GitHub Actions. It's important to use secrets here and not commit these values directly to the Git repository (as they would be unprotected and would be accessible by any person with access to the repository then). Make sure to define these secrets in GitHub from the repository settings.

When you commit this workflow config to GitHub, and if you have defined the secrets correctly, GitHub Actions will run our workflow and execute our test suite three times in parallel, once for each browser:

Running our Sauce Labs tests with GitHub Actions (with parallel test execution)

Because we called our test-ci script alias in our CI pipeline, which also uses and runs our testmo tool, the three test runs are also automatically reported to Testmo. We can also see the full console output, test times, exit codes and all test results in Testmo after the CI run finished. For example, a test automation run in Testmo looks like this:

Sauce Labs reporting of test runs to Testmo from GitHub Actions

That's it! Starting with our initial automation script we've developed our Selenium test automation suite, implemented Sauce Labs reporting, submitting our test runs to test management and finally integrated our test suite with a GitHub Actions CI pipeline.

If you enjoyed this article, also make sure to subscribe to notifications about our upcoming articles that will cover similar integrations with more testing services, frameworks and dev tools.

Start Testing with Testmo Free

#1 Unified Modern Test Management Software

Start Free Trial See All Features

PS: We regularly publish original software testing & QA research, including free guides, reports and news. To receive our next postings, you can subscribe to updates. You can also follow us on Twitter and Linkedin.

Sauce Labs Reporting & Selenium Test Results With Testmo

Initial Project & Repository Setup

First Sauce Labs Browser Automation

Running Our Code Against Sauce Labs

Complete Browser Test Automation Suite

Reporting Results to Test Management

CI Pipeline Integration (GitHub Actions)

Start Testing with Testmo Free

Start testing with Testmo for free today