GitLab CI/CD makes it easy to run automated tests against your code. Once you have written your test suites, you can configure jobs in your GitLab CI/CD pipelines to do things like build your app, carry out automated tests on it, and report the results to a test management tool like Testmo.

Out of the box, GitLab CI/CD gives you the ability to schedule pipelines to run at regular intervals, trigger them automatically (based on code commits, merge requests, API calls, webhooks etc.), or launch them manually from within the GitLab CI/CD platform.

In this guide, we will walk through a detailed example of how you can trigger test automation pipelines right from within Testmo. Launching automation jobs from Testmo gives the QA team the power to launch automated tests at any time, without having to switch tools or wait on developers or the DevOps team. It also gives you the ability to launch test automation more quickly and improve your visibility over all of your testing, including your manual, exploratory, and automated tests.

If you are not yet familiar with basic test automation with GitLab CI/CD pipelines or how to run automated tests, we have  created separate detailed guides on this. We recommend reading these articles first if the concepts are completely new to you, as we will not cover the basics in this article:

• GitLab CI/CD Parallel Test Automation Jobs
• GitLab CI/CD Test Automation Pipeline

Setting Up Your GitLab Pipeline

For this article and example project we have set up a GitLab repository, building on our previous GitLab CI/CD Parallel Test Automation Jobs example. You can review the files in the GitLab repo and use them to follow along with this example in your own GitLab account.

Before we can trigger GitLab CI/CD pipelines from Testmo, we need to configure them in GitLab. In our example, we have set up our pipeline to execute the following jobs:

1. build - a dummy code build step, for demonstration purposes
2. test-setup - creates an automation run in Testmo, ready for reporting results to
3. test - executes the tests, in parallel threads
4. test-complete - submits the test results to Testmo
5. deploy - deploys the dummy code to our specified target environment, with the requested dataset

We’ve made a couple of changes to the .gitlab_ci.yml file however, to accommodate how we expect to use our pipeline for the following scenario:

As a tester
I want to build and deploy code to a specified environment on demand, from Testmo
And include a testing dataset
So I can perform my testing on a smoke-tested and pre-staged environment

To be able to select options for deploying to different environments or using specific datasets, we can specify variables in our .gitlab_ci.yml file, as shown below:

# variables:
variables:
  ENVIRONMENT:
    value: "test"
    options:
      - "production"
      - "staging"
      - "test"
    description: "Which environment to deploy to; Defaults to test."
  DATASET:
    value: "none"
    options:
      - "small"
      - "large"
      - "none"
    description: "Which dataset to use; Defaults to none."

The variables tell GitLab that an environment must be specified, and provides a default value of test along with valid options of staging or production. We also specify that a dataset value should be provided. Valid inputs for the dataset option are small, large or none (for the event that no data is required, like when deploying to production). Once the repository has been initialized with the workflow and the necessary testing files (explained in our guide on parallel testing in GitLab), starting the job in GitLab CI/CD directly will require a confirmation of the inputs before executing the pipeline:

Before going ahead and triggering the job, we should add the TESTMO_URL & TESTMO_TOKEN secrets to the GitLab CI/CD settings, since the test-setup and test-completesteps will require those in order to create our run in Testmo, and to submit the test results. 

You can configure these by going to Settings > CI/CD in your GitLab project and adding two new variables called TESTMO_URL and TESTMO_TOKEN with the relevant values. Both variables should be configured as Protected and Masked, so they will also be hidden in any console logs. These variables are then automatically available as environment variables during our CI pipeline run.

test-setup:
  stage: test-setup
  script:
    - npm ci --cache .npm --prefer-offline
    # Optionally add a couple of fields such as the git hash and link to the build
    - npx testmo automation:resources:add-field --name git --type string --value $CI_COMMIT_SHORT_SHA --resources resources.json
    - npx testmo automation:resources:add-link --name build --url $CI_PIPELINE_URL --resources resources.json
    # Check if the required variables are available
    - >
      if [[ -z "${TESTMO_URL}" ]]; then
        echo "The TESTMO_URL secret is not defined for this repository"
        exit 1
      fi
      if [[ -z "${TESTMO_TOKEN}" ]]; then
        echo "The TESTMO_TOKEN secret is not defined for this repository"
        exit 1
      fi
    # Create test run
    - npx testmo automation:run:create --instance "$TESTMO_URL" --project-id 5 --name "Parallel mocha test run" --resources resources.json --source "unit-tests" > testmo-run-id.txt
    - echo "TESTMO_RUN_ID=$(cat testmo-run-id.txt)" > testmo.env
  artifacts:
    reports:
      dotenv: testmo.env

Your GitLab pipeline should now complete successfully. When it does, you’ll be rewarded with a view like the one below. All the steps are green! Now you’re ready to launch the job from Testmo.

Configuring Testmo to Point to Your GitLab Pipeline

Now that you have set up your GitLab CI/CD pipeline to run some demo automated tests and report the results to Testmo, we can look at how to launch that pipeline from Testmo.

Connect Testmo to GitLab

The next step is to connect Testmo to your GitLab account, if you haven't done so already. The Testmo GitLab integration relies on a secure OAuth-based connection, which you can enable via the Admin > Integration area in your Testmo account. To see how set up a connection from Testmo to your GitLab account, you can follow our documentation.

Once you have successfully connected Testmo and GitLab, you can come back to this article and continue to the next step!

Set Up an Automation Target in Testmo

With the GitLab integration completed and connection successful, we can add our GitLab pipeline as an automation target in Testmo. Automation targets give you a way to specify pipelines, branches, or tags that you would like to be able to trigger ahead of time. While setting up automation targets, you can also include unique parameters, inputs, or variables to trigger pipelines for particular environments, datasets, etc.

For this example, you can set up your automation target in Testmo by following the steps below:

• In the Admin > Automation view in Testmo, click on the Add Target button.
• In the Add target dialogue:
  • Name: Give your target a suitably descriptive name. To reflect our scenario, let's enter something like “smoke test & deploy code with data”.
  • Connection: Select the GitLab connection you just configured in the previous step as the CI/CD connection.
  • Settings > Project Path or ID: Add the GitLab group and name of your project (or the project ID) from GitLab. The group name and project in which you’ve implemented the pipeline example from above, for example: “yourgroupname/example-automation-launching”.
  • Settings > Reference: Add the Reference branch or tag name to be used in the pipeline. We just have a main branch in the example above, so we can just add “main” here.
  • Variables: Add any variables you wish to supply to GitLab as part of the workflow. 
    • In order to satisfy our environment and data requirements, we can specify the inputs to the workflow here by adding them to our target:
      • DATASET = small (or large)
      • ENVIRONMENT = staging (or production)
• Once you’ve added all the settings & inputs, then click Add target save the target.

As an additional step, you may wish to constrain the target to only a specific project(s), preventing it from being triggered from unauthorized projects. If so, select the desired projects from the Projects tab in the Add target dialogue.

Triggering the GitLab Pipeline from Testmo

Once you’ve added the target, you can go to your project(s) and launch your test automation pipeline by adding an Automation job. Simply navigate to the Automation > Jobs view in any project where you enabled your GitLab target and click the Add Job button. Choose the target you just configured, verify the settings, and click Add job:

Testmo will queue your job for execution, and once it has triggered your GitLab CI/CD pipeline, let you know whether the it was launched successfully, or not. 

• If it the job launched successfully, you’ll see a blue cog. Clicking on the link will open the job page directly in GitLab, so you can see the progress and outcome there.
• If the job wasn’t triggered successfully, you’ll see an yellow exclamation mark instead. Clicking on the details link will provide a summary of the failure reason. Often the failure will be down to a misconfiguration of the target, or a missing or unexpected input. Our FAQ’s page contains helpful advice for troubleshooting failures.

Viewing Automated Test Results in Testmo

Assuming the job was executed successfully, the tests will also have been run and the results will be reported to the specified Testmo instance (remember the TESTMO_URL & TESTMO_TOKEN secrets from earlier!) for review. You’ll be able to see what happened with them in detail in the Automation > Runs & Results view.

By drilling down to the specific run, you’ll also be able to see valuable information about the test executions like how long they took to run, which thread contained what tests, what the slowest tests were, whether there were any failures, what CI job generated the tests, and much more besides. You can also download a report of your test results and share it with interested stakeholders by simply hitting the print button. 

Conclusion

That’s it! We’ve successfully set up a GitLab CI/CD pipeline so that it builds our code, runs some smoke tests against it, sends the results to Testmo and deploys the code to our specified environment with the desired dataset. We’ve also integrated GitLab with Testmo so that the pipeline can be launched on-demand from Testmo, giving you the power to launch your automated tests at any time, right from Testmo, and leaving the deployment and test automation to GitLab and your chosen test framework (Mocha, in our example above).

If you want to learn more about using Testmo with GitLab and test automation tools such as Selenium, we have lots of other articles you can read too: 

• Product: GitLab integration features
• Guide: GitLab CI/CD Test Automation Pipeline & Reporting
• Guide: GitLab CI/CD Parallel Test Automation Jobs
• Code: GitLab CI/CD test automation example
• Code: GitLab CI/CD parallel test automation example
• Docs: Testmo CI/CD integration
• Docs: Testmo CLI tool
• Docs: Testmo test automation

 

PS: We regularly publish original software testing & QA research, including free guides, reports and news. To receive our next postings, you can subscribe to updates. You can also follow us on Twitter and Linkedin.