We’re excited to announce the availability of a new report within the Testmo Reporting Center: the Automation Metrics report!

This new report is designed to help you report on testing activities and results associated with your test automation, enabling you to see automation trends over time and share key information with stakeholders regarding your test automation results.

What is the Automation Metrics Report?

The Automation Metrics report allows you to generate summary statistics and detailed breakdowns of all the automated testing activities carried out across your automation sources under a single project.

Once your automation results are uploaded to Testmo, they get aggregated by source for ease of visibility. The Automation Metrics report enables you to consolidate information from multiple sources into a set of high level charts, lower level statistics and detailed per automation run breakdowns, to a single view which can easily be digested by stakeholders across your team.

The report focuses on automation runs which are grouped primarily by sources. Automation runs can also be linked to milestones, which can be viewed alongside manual test case runs and sessions in our Milestone Summary & Metrics report.

See our documentation for the Reporting Center here:

• Reporting center overview
• Requirements coverage & traceability report
• Milestone summary & metrics report
• Automation metrics report

Key Capabilities of the Milestone Automation Metrics Report

This report offers several key capabilities to give you deep insights into your milestone-based testing.

• Unified Test Automation View: Aggregate and report on all automation runs in a single project.
• Comprehensive Scope Control: Easily select a single project and one or multiple automation sources for your report.
• Flexible Configuration: Tailor the report’s content to your needs by including or omitting sections like the automation run timeline chart, automation metrics, statistics and detailed automation run breakdowns in card or table formats.
• Filtering: Filter automation runs based on criteria like status (passed, failed) or custom filter settings. Limit the total number of items displayed using the maximum limits.
• Customizable Presentation: Control the report’s appearance by enabling or disabling elements like links back to Testmo, personal names/avatars, and the report header and footer.
• Easy Export and Sharing: Export the generated report to PDF for easy sharing with team members or stakeholders.

Example: measuring automation velocity across sprints

You can follow the example steps below to create a report which shows the progression of your automated test metrics over a period of time (e.g. a sprint).

Creating the report

On the Automation metrics report page:

• Select your project.
• Apply a source filter so it constrains the report to only the UI tests (or tests of interest if something other than the UI, e.g. API tests).
• Select the Runs > Filter runs option and apply a custom date range matching the start & end dates of the sprints you want to compare across (e.g. a 4 week period covering the start and end dates of your last 2 sprints).

When applying a custom date range the Time frame field will automatically be set to All time, enabling the custom period to override the setting. Changing it to another time frame will override the custom duration. You can revert to the custom duration by setting the Time frame option back to All time.

• Set the Group by option to Day (the default option).
• Click the Generate button.

Information about your automation tests can be derived from various places on the report:

• The Run Timeline by Day chart will show how many automation runs were executed each day for the selected, period grouped by whether the runs passed or failed.
• The Run Metrics by Day chart will show the average number of tests per run, per day, along with their average durations and thread counts. We’re looking for an upward trend in the average test count here.
• The Automation Statistics chart will should give us a good idea of the answer to the question we’re asking. The Tests per run count should be increasing over the period. We can also see from the Tests per run +/- value whether the number of tests has increased (in green) or decreased (in red).

Summary & What’s Next

This update is now live for all Business & Enterprise Testmo accounts, so you can start using the new features right away. If you aren’t using Testmo yet, you can also register a free Testmo trial to get started.

We are also already working on various other new features and improvements for Testmo. We have many new things planned for this year, so stay tuned. If you have an idea for an improvement or new feature you would like to see in Testmo, please let us know!