Key Software Test Automation Values
There is nothing surprising nor new in this blog title. A high degree of software test automation is key for mature DevOps workflow through continuous testing. The issue at hand is that maturing test automation on its own is easier said than done, and there are various reasons to why that’s the case.
In a cool workshop, I delivered as part of the Quest2018 conference in San Antonio, I was able to validate that majority of test engineers, test managers, and developers are fully aligned with the below “excuses” as to why their test automation processes aren’t maturing. Some have a combined list of pains from the below, while others can relate to the entire list.
Building blocks towards automation that works
To resolve many of the above challenges, organizations need to resolve the following:
- Rely on a robust test framework that fits both the product technical requirements as well as the teams’ skills
- Stabilize their CI and test automation code, and only then add more new tests to the suite
- Have a solid test lab (most recommended is a cloud-based) that is always available, integrated into the teams’ toolchain, secure, and that can scale as needed
- Treat test automation code as production code with all the meanings
- Follow test automation practices for resolving flakiness, as well as making the test reusable
- Choose the right software delivery methodology for your organizations – ATDD, BDD, etc.
- Not every test needs to be automated. Use proper judgment and pick the high value, stable test scenarios that will be part of your continuous testing and CI
Why automate at all?
To achieve the above goals and have high coverage of automation is hard. Is it worth it?
The short answer to the questions is – as a modern software company, you do not have much choice. To be agile and deliver value fast to your customers you need a continuous process that is as less manual as possible that can fit a short iteration of 2-3 weeks or even less.
For any IT company that develops in SCRUM need the ability to test different versions of the system simultaneously and run extended test coverage – run sanity tests for example on the live version of the system while running regression on the version in development – a job that can only be done with the use of automation.
If you put the test automation rationale into sections, it would look as follows:
- Velocity enablement
- Robustness and reliability
- Responding to changes – Market and Product
- Audit trail and quality evidence
- Extended test coverage – scenarios that cannot be done manually
There are most likely additional benefits to test automation, but in this post, I will review and focus on the above.
There is no doubt that executing at scale is by far faster when done by a machine vs. human.
When developing software through an iterative process of 2-3 weeks, being able to execute new test code alongside regression is an integral part of the release criteria and the definition of one. Imagine doing all these activities manually, this will obviously slow down the entire release process. Also, test automation does not require (when implemented properly) human intervention, which entails a 24/7 operation.
Source: Software Testing Genius
Robustness & reliability
Manual testing is error-prone, and indeterministic compared to consistent machine-based test automation. Being able to consistently get the same results when the expected criteria are met, or not is key to release high-quality products. Such reliability at scale and as the product grows is a critical enabler for continuous testing.
Responding to changes
Most organizations are part of the digital ecosystem, hence, exposed to recurring market events like new operating system release, new device launch, and more. In addition, as part of the benefits of Agile and DevOps, changing product requirements based on business objectives and customer requests is part of the routine both Dev and Test face on a weekly basis – dealing with these changes in an efficient way can only be achieved by having a robust test automation foundation in place that can be adjusted and triggered on-demand as changes occur.
Audit Trail and Evidence
Test automation leaves trails and can be retained for as long as needed, while manual testing is by nature harder to record and store. Specifically, having the test data for ad-hoc audits, and compliance processes is a clear business requirement for verticals like healthcare and insurance.
Extended test coverage
Test automation can surpass manual testing with test types like visual validation, performance and load testing, security and code analysis, production monitoring, cross-environment testing, execution at scale, and many more. In today’s testing reality there are no longer naïve test scenarios, but rather complex and cross-environment dependent test cases that require a sophisticated lab and test framework to support these requirements.
The case for having software test automation as a key enabler for the entire DevOps and business success was proven years ago, and in reality, this case has grown in significance due to the advanced technologies. It is now all about expanding test automation capabilities through best practices, robust test lab, and high-value test frameworks to support the digital changes.