Automation Fields, Updated Testmo CLI & Test Steps

Today we are announcing an update for Testmo and our CLI tool to make it easier to include useful additional information with test automation results such as custom fields, test steps and links to external artifacts (such as logs or screenshots).

Testmo already supports fields for test automation results, but not all existing automation tools make it easy to include these in the standard XML result files. So we are extending Testmo's CLI tool to detect & read fields from test outputs, we are introducing support for automation field type hints, we are adding additional rendering options for test steps, and we are publishing new documentation on how to use these features with Testmo.

Ready to try the new features?

Make sure to update your Testmo CLI tool to the latest version (at least 1.2) to benefit from all the new features. If you are always automatically installing the latest version within your CI pipeline, you don't need to do anything.

Tracking & reporting test automation runs with Testmo

Using Fields with Automation

Adding additional fields and meta information directly to your test automation code and submitting them together with your test results to Testmo has many advantages.

Many test management tools require you to manually create and maintain automated test cases in their tools in addition to building your tests in code. We at Testmo think this is unnecessary extra work and there's a better approach. Teams only want to maintain their automated tests in one place (in code), and then submit the results to Testmo to have all their automation results (from all their different test suites) in one place. Testmo provides a single source of truth for all your test automation suites for full test automation management & reporting.

Maintaining automated tests with extra fields and meta information directly in your automation code (opposed to maintaining them separately with manually created test cases) and then reporting them to Testmo has many benefits:

Maintaining tests in one place only: You only need to maintain your automated tests in your code, so you can easily & quickly update your tests, fields and steps in one place. There's no need to also manually update your automated test cases in Testmo, which many tools require & which is error-prone and usually results in out of date test cases.
Single source of truth in Testmo: With Testmo you can easily submit all your automated tests with any tool and from any platform. So Testmo is your single source of truth for all your test automation suites, including UI/frontend tests, API, unit, performance, mobile & backend tests. If you add, update or remove tests from your automation suites, Testmo always has your latest test results and is always up-to-date and is never out of sync.
Automatic versioning & branches: Software teams often work on multiple versions, releases and branches at the same time. Many teams work on longer-term features in separate branches, or need to quickly fix and test an issue customers found in production. With your automated tests & fields maintained together in code, your tests are automatically versioned and support branches just like your application code.
Accurate & scalable history: If you maintain & enter your test automation code and fields/meta information separately, you are never sure if your past test results actually match the test details you maintained in your test management tool at that time. With Testmo, you can always be sure that the test results & details accurately match what you tested. Testmo allows you to scale your test automation suites with less maintenance work & with full confidence.

Automation results with additional fields, links, custom rich HTML, steps & more

There are various fields and extra information that can be useful to include with your automated tests. For example, you might want to include fields that document what a test is actually verifying, or include a test description, author or severity. It can also be useful to include short source code snippets such as Gherkin feature files with rich syntax highlighting.

Or you might want to include additional details such as links to screenshots, logs or build artifacts etc. that make it easier to understand why a test failed. Let's take a look at how to include and submit additional fields for your automated tests to Testmo.

How to Send Automation Fields

Different test automation tools support different ways to include additional fields for your tests. Ideally, test automation tools directly allow you to specify additional properties for your tests that are then added to the resulting XML files. For tools that don't support this yet, we've also implemented support for an alternative approach to parse test case properties from test outputs.

Option 1: Test Case Properties in XML

If your test automation tool supports specifying additional test case properties, then this is the easiest way to get your fields submitted to Testmo. For example, tools such as Pytest or Playwright have native support for this. More and more other tools are also adding support for test case properties, and we are working with various open source automation tools to get this implemented. Our CLI tool supports various formats for this:

<testcase name="testNewUser" classname="Tests.Registration" time="3.441">
    <properties>
        <property name="severity" value="high" />
        <property name="language" value="english" />
        <property name="author" value="Adrian" />
        <property name="attachment" value="https://ci/files/22-dashboard.png" />
        <property name="attachment" value="https://ci/files/23-users.png" />

        <!-- Optional support for properties with text values -->
        <property name="description"><![CDATA[
            This text describes the purpose of this test case and provides
            an overview of what the test does and how it works.
        ]]></property>
    </properties>

    <!-- Optional output or error/failure/skipped -->
</testcase>

Including test case properties in the XML file (see more examples)

Option 2: From Test Cases Outputs

If you are using a test automation tool that doesn't support test case properties yet, we are also supporting an alternative approach to get these submitted to Testmo. Many automation tools allow you to capture the test output during execution and write the output to the resulting XML files. Our CLI tool now implements a syntax to parse and read properties from this test output as an alternative way to submit fields. Various automation tools support this and we have updated our documentation with examples for Node.js Mocha, Cypress, Nightwatch.js and PHPUnit.

<testcase name="testNewUser" classname="Tests.Registration" time="3.441">
    <!-- If you can write test output to the XML file, you can 
    include fields for Testmo -->
    <system-out><![CDATA[
Output line #1
Output line #2

[[PROPERTY|author=Adrian]]
[[PROPERTY|language=english]]

[[PROPERTY|browser-log]]
Log line #1
Log line #2
Log line #3
[[/PROPERTY]]
    ]]></system-out>
</testcase>

Including test case properties in test output (see more examples)

Option 3: Sending Fields with Runs & Threads

Last but not least, Testmo also supports sending additional custom fields, links and artifact references for entire test runs and automation threads. This doesn't depend on any automation tool support so this works with any test automation suite. You simply use our CLI tool to create a resource file with any fields and references you want to include, such as links to the CI pipeline run, version numbers, Git commit hash, screenshot store etc. And you can then submit these resources with any automation run or thread. Please also refer to our documentation on submitting run & thread fields with Testmo.

$ testmo automation:resources:add-link \
  --name "GitHub repository" \
  --url "https://github.com/my-org/my-repo" \
  --note "Our project's git repository with all code files and issues."

$ testmo automation:resources:add-field \
  --type "string" \
  --name "Version" \
  --value "2.3.1-5fbcc8d0"

# For text, console and HTML fields, make sure to pipe the
# field value via stdin, e.g. from a file:

$ cat files.txt | testmo automation:resources:add-field \
  --type "console" \
  --name "File list"

# The resource file can then be used with any command via the --resources option

Submitting fields and links for automation runs & threads (see more examples)

Including additional fields, references and artifact links for automation runs & threads

Automation Field Types

Testmo supports various value types for additional fields you include with your automated tests. You do not necessarily need to specify types, and Testmo will simply render the fields as string or text values then. But if you would like to format your additional fields in a certain way, then it can be useful to include type hints with your properties to tell Testmo how to render these fields. For example, you can tell Testmo that a property includes an URL, so Testmo can render this as a clickable link. Or you can render HTML to use rich formatting such as tables, lists, test steps etc. Testmo currently supports the following types:

string - The default for single line fields
text - The default for multi-line fields
url - Renders a clickable link
console - Renders as monospaced console/output text
html - Renders the value as HTML (reference)

Type hints are specified as part of the property names by specifying the type before the name, separated by colon. Testmo also automatically detects and removes the type from the name then. Here are some examples:

<properties>
    <property name="string:severity" value="high" />
    <property name="url:github" value="https://github.com/project/code.js" />
    <property name="text:description" value="Long text field.." />
    <property name="console:log"><![CDATA[
        Log line #1
        Log line #2
        Log line #3
    ]]></property>
    <property name="html:richtext"><![CDATA[
        <h1>Title</h1>
        <p>Text</p>
    ]]></property>
</properties>

Testmo supports type hints to render fields as strings, links, console or rich-text HTML

[[PROPERTY|string:severity=high]]
[[PROPERTY|url:github=https://github.com/project/code.js]]
[[PROPERTY|text:description=Long text field..]]

[[PROPERTY|console:log]]
Log line #1
Log line #2
Log line #3
[[/PROPERTY]]

[[PROPERTY|html:richtext]]
<h1>Title</h1>
<p>Text</p>
[[/PROPERTY]]

You can use type hints the same way for properties included in test outputs

Automation fields support various field types such as strings, URLs, HTML & console

Test Steps & Attachments

Testmo now also understands and supports some common fields with extra rendering features. You can now include test steps with your automated tests and Testmo renders them just like your manual test case steps when viewing your automation runs. And Testmo supports links to externally hosted attachments & test artifacts now, so you can include links to screenshots, log files, error reports and more.

Automation Test Steps

To include and render test steps for your automated tests, simply include properties named step* with your tests. Testmo will then render these steps for your test runs. You can optionally also include a test result status for each step, as well as optional sub step fields. You can either include multiple properties with the same step name. Or if your automation tool's API doesn't allow this, you can also add a unique index to the fields and Testmo will automatically understand this (e.g. step1, step2 etc.).

<properties>
    <!-- Either use multiple properties named "step" -->
    <property name="step" value="This is the first step." />
    <property name="step" value="And this is the second step." />

    <!-- Or append a unique index if your test framework requires this -->
    <property name="step1" value="Testmo will automatically index the steps." />
    <property name="step2" value="And this is another test step." />

    <!-- Some tools support multi-line properties. Steps always support HTML -->
    <property name="step"><![CDATA[
        This is a more complex test step with <strong>multiple lines</strong>
        of text using a property with a text value.
    ]]></property>

    <!-- You can optionally also include a step result status as part
    of the property name in square brackets. All status automation 
    aliases as configured in Testmo under Admin > Statuses work. -->
    <property name="step[passed]" value="This is a test step that passed." />
    <property name="step[failure]" value="And this step had a failure." />
    <property name="step[skipped]" value="This step wasn't executed." />
    <property name="step[error]" value="And this step had an error." />

    <!-- Testmo also supports step sub fields using the following special tags. -->
    <property name="step[passed]"><![CDATA[
        <test-step-subfield name="Step">
            This line describes this step. You can also use <strong>HTML</strong>.
        </test-step-subfield>
        <test-step-subfield name="Expected">
            You can optionally include more sub fields, e.g. for the expected results.
        </test-step-subfield>
    ]]></property>
</properties>

Optionally include automation steps (see more examples, including steps in test outputs)

Links to Attachments & Artifacts

Testmo also supports multiple options to include links to external attachments such as screenshots, log files and test artifacts. Simply include properties named attachment or, if your tool doesn't support multiple properties with the same name, you can add a unique index like attachment1, attachment2 etc. Please note: Attachments need to be URLs / links to externally hosted files, e.g. in your CI pipeline service. Additionally, Testmo also supports the standard [[ATTACHMENT]] syntax for test outputs as used by various tools and you can learn more in our documentation.

<properties>
    <!-- Either use multiple properties named "attachment" -->
    <property name="attachment" value="https://ci/files/22-dashboard.png" />
    <property name="attachment" value="https://ci/files/23-users.png" />

    <!-- Or append a unique index if your test framework requires this -->
    <property name="attachment1" value="https://ci/files/24-sessions.png" />
    <property name="attachment2" value="https://ci/files/25-failure.png" />
</properties>

Optionally include attachment links (see more examples, including in test outputs)

Steps (with optional statuses and sub fields) and artifact links for automation results

Integrations & Building Tools for Testmo

Today we are also launching various additional topics and references as part of our documentation to make it easier to use automation fields, our new XML & HTML references, as well as our guide on how to build and extend tools that work well with Testmo.

Test automation fields: We have published this extensive guide on our new support for automation fields and properties. It covers all topics such as supported field types, conventions and syntax.
Testmo XML reference: To make it easier to see all tags, attributes, properties, field names and conventions supported by Testmo, we've published a full XML example and reference.
Testmo HTML reference: We support rich fields with HTML support in Testmo. This allows you to use rich-text formatting, include tables, lists, image references and more. We have published this HTML reference with all supported HTML tags and CSS classes.
Contribution guide: Many tool maintainers and vendors are interested in adding additional support to their automation tools optimized for Testmo. We have published this guide with useful tips on optimizing tools and open source projects for Testmo.
JUnit XML format: JUnit-like XML files have become the standard format to exchange test automation results between tools. We have published a new repository on GitHub with documentation, references and examples for testers and developers interested in learning more about this format.

Summary & What's Next

This update is now live and we have published our new CLI tool version. The new features are now also available in all Testmo accounts. To use the new CLI features, make sure to update the CLI tool to the latest version (at least 1.2). If you aren't using Testmo yet, you can also register a free Testmo trial to get started.

We are also already working on various other new features and improvements for Testmo. We have many new things planned for this year, so stay tuned. If you have an idea for an improvement or new feature you would like to see in Testmo, please let us know!

Start Testing with Testmo Free

#1 Unified Modern Test Management Software

Start Free Trial See All Features

PS: We regularly publish original software testing & QA research, including free guides, reports and news. To receive our next postings, you can subscribe to updates. You can also follow us on Twitter and Linkedin.