Measuring test automation efficiency, its ROI (return on investment), and value to the overall quality of a product is one of the common topics in test automation. For years, managers, practitioners, and agile teams are trying to justify and measure test automation. This topic is still relevant when connecting it to continuous testing and DevOps, but a bit different than before.
In the past, teams were trying to measure test automation around the percentage of test cases that are automatable, and the cost savings associated with this task. When putting test automation in the context of DevOps, the driver for measuring test automation and testing, in general, should be the VALUE of the tests and their ability to mitigate and reduce quality risks.
Continuous Test Automation and Different Practitioners
Before covering ways to measure test automation effectiveness, let’s do a short recap of the personas that test automation is serving.
Test automation must be measured not just by automating functional testing by SDETs and/or business testers. Developers within feature teams also take an active part in test automation, hence, such activity needs to consider all practitioners.
This means that test automation should have proper metrics that cover different test tools that these personas use daily to achieve their quality as well as their personal goals.
The above is extremely important since if there is a wrong match between the persona and the tool they use to create test automation scripts, the measurements are in a sense irrelevant.
In addition, the usage of the tools by the practitioners need to also be measured in the context of the test pyramid according to the project’s complexity and needs. Testing UI/API, Functional, Integration, Non-Functional aspects (load, accessibility, etc.) and unit testing need to be measured accordingly.
When considering the ROI of testing, and test automation specifically, executive and managers need to consider the risks of lack of quality, escaped defects, late releases, etc. rather than just the cost of a software license. This does not imply that there is no need for proper due diligence, considerations, and matching between the test automation and the job to be done.
Lastly, the type of application in many cases will determine a lot of metrics, tools, and considerations teams will take: Web vs. Mobile vs. IoT vs. other or mix of apps has a great impact on many of the test automation metrics.
Measuring Test Automation ROI – Key Recommendations and Metrics
After establishing the ground for measuring automation by defining personas, application type, test types and the difference between test automation and test automation in DevOps, let’s focus on measuring and defining solid and high-value test automation.
Since test automation within continuous testing is something teams execute multiple times a day, per each code commit, within CI, and expect value, fast and actionable feedback from – therefore, putting extra emphasis on the creation of the tests is a strong recommendation.
As in the below sketch, three pillars of good test automation include the high value of the tests, high reliability, and fast execution and feedback. If these are part of the pillars, some of the metrics of how to measure test automation ROI must be around these points.
When determining the ROI of test automation the process stages also need to be part of the math:
- Test authoring (time to create a test, ease of use, script reusability)
- Test execution abilities (parallel testing, connectivity, and integration with standard tools and frameworks)
- Test results analysis
- Test maintenance simplicity
- Software license cost per user/floating
- Number of resources required for the above
Generic Continuous Testing in DevOps List of Metrics
- How quickly are testing activities moving, and what is slowing down these activities?
- Test flakiness.
- Test duration.
- Percentage of automated vs. manual tests.
- Application quality measurements
- Number of escaped defects and in which areas.
- MTTD — mean time to detection of the defect.
- Build quality.
- Pipeline efficiency measurements
- The number of user stories implemented per iteration.
- Test automation as part of DoD across iterations.
- Broken builds with categories.
- CI length trending.
- Lab availability and utilization.
- Quality cost measurements
- Operational costs, lab availability issues.
- Cost of hardware/software.
- Costs of defects by severity and stage
Finally, as an important reminder, there is the common formula of ROI measurement:
ROI = Gain – Investment/Investment
If to use the above formula and refer to an old but still relevant blog from TestingWhiz, and make some placements according to some key measures (feel free to configure based on your own project needs), we can see the following: The gain will be the objectives and goals you wish to receive from test automation – reduction in feedback, shorten cycles, release of manual engineers, converted into $$, and so on.
Measuring test automation ROI should focus on more than just licenses cost and reducing testing cycles. It is a mix of the value, reusability, analysis time, and matching to the project, persona, and technology teams are using.
The recommendation is to come up with a mix of the above, place some abstract “grades” per each of these items, and prioritize them to reach the answer.
From a great practice I received from my friends at USAA, I can also recommend the following formula to choose the right tool for the project.
The above recommends choosing the tools using a set of considerations from both test development practice (BDD, ATDD, etc.), type of testing E2E/Visual, and the ease of use that includes also documentation. Each consideration gets a weight and scoring key that when multiplying both, you get a result that can guide your selection.
Happy Test Automation!