5 questions to ask when deciding which test cases to automate

 In Blog, Most Recent, Continuous & Regression Testing, Web Application Testing, Agile & DevOps Testing, Manual Testing

5 questions to ask when deciding which test cases to automate

You’ve heard the hype from your friends and colleagues: test automation is the way to go if you want your QA team to keep up with your company’s frequent release cycles. It comes with many benefits, including improving your testing efficiency, increasing your test coverage, and detecting bugs earlier.

Cost of automation testing vs. manual testing

Source: Medium

Yet whether you’re implementing test automation for the first time or are looking for new ways to scale your automation in the new year, choosing which test cases to automate can feel like a daunting task. To help, here are some important questions to ask when making this decision as part of your larger test automation strategy:

How valuable is my test case?

The first place to start by looking at your users, and determine which types of actions they do most frequently with your application. Since these are the areas where your users are focusing most, their tests stand to provide the most value when you automate them.

Once you have identified these use cases, you can look within their respective test cases and see whether automation can help reduce your QA team’s time and resources. By focusing on these test cases more closely, your team will be able to manage expectations and set relevant KPIs accordingly.

Another way to ask this question is whether or not automating this test case will support a greater business goal. For example, creating an automated test that covers attributing discount codes to customer purchases can make a positive impact on a business that is looking to increase overall revenue due to site-wide sales. This will ensure that your end-users enjoy a seamless user experience while interacting with your application, plus the right test automation tool will alert your team more quickly if there are any bugs. Plus, if you’re just starting with test automation, aligning your test cases with larger business goals is always a good strategy to convince your management team that test automation is a worthwhile investment.

How often does this test case need to run?

When starting your test automation journey, many companies also find it helpful to start with test cases that run frequently while also being monotonous or repetitive. These test cases are often simpler in nature, but still necessary to run on a frequent basis. By automating these types of test cases, testers will benefit most in terms of saving time and avoiding unnecessary human error from running a basic test one too many times.

While focusing on simple, yet time-consuming test cases is a great place to start, don’t think that this is all that’s available when it comes to automation testing. More complex test cases that run often are fair game for automation as well (and in fact, recommended), but it’s important to determine whether they happen frequently enough or are valuable enough to merit automation.

One example of a complex test case that can be particularly helpful to automate is ensuring when to give eCommerce customers free shipping. Many eCommerce sites offer customers free shipping when they surpass a certain spending threshold, or only support free shipping to certain countries. Yet while creating automated tests for shipping processes can be identified as a complex test case, companies will want to automate since shipping is a high-value process that happens often.

How much risk is involved?

Another important question to ask when deciding which test cases to automate is the risk involved. When considering the different tests that you need to run for your application to run properly, it’s also critical to think about what would happen if those tests were to break.

In any type of manual test, there’s a greater chance of mistakes simply due to human error. Especially when it’s a test that’s repeated often, any minor lapse in testing has the potential to cause failures that can be catastrophic for your business. If the test case in question will either have a significant negative impact on your users or your internal team’s downtime if it breaks, it’s likely a relevant test case for automation.

How much does this test case currently cost?

When thinking about the cost of running a test, there are different factors that you need to take into account. Some of these include:

  • Time spent on running the test case manually
  • The time it takes away from doing other tasks
  • Test maintenance
  • Resolving bugs

Each of these tasks takes time away from your manual or business testers, shortening their bandwidth for them to focus on other, potentially more pressing tasks. In addition, having your business testers spend their time doing repetitive, monotonous work is also a drain on their workplace morale and can contribute to high QA turnover. Therefore, another cost to factor into running tests is hiring and onboarding new QA’s on a frequent basis to bring them up to speed on your manual testing processes.

If your test case in question is proving to be an unnecessary drain on your QA team (and packing some hidden costs for the company), it may be worthwhile to automate. While any type of test automation requires some sort of learning curve, it has the potential to reduce your costs in the long run when you choose the right test case.

Does it fall within the test automation pyramid?

On a more technical note, another good rule of thumb is to consult with Mike Cohn’s test automation pyramid when deciding which test cases to automate. In the pyramid, Cohn highlights which test cases to focus on more for automation, and which types of tests to automate more infrequently.

The bottom layer of the pyramid focuses on unit tests, or tests that focus on small, specific units of code. The reason why Cohn recommends unit tests for most of your automation is that they are extremely fast to execute and easy to troubleshoot when there’s a failure. You can also run unit tests after every code build, which will give your team quick feedback after doing regression testing.

The middle layer is comprised of service tests, such as API tests and acceptance tests. They allow for testing the business logic of various processes without actually doing UI testing. Cohn attributes these to the middle layer of the pyramid since they are still faster and more reliable than UI tests since they don’t need to deal with frequent changes.

He then allocates the top layer to UI testing, since it’s a much smaller area to work on when most of the code and business logic has already been tested. Additionally, UI tests are typically more brittle (since the UI changes so often) than unit tests and service tests, so it generally takes more effort to get ROI from this type of testing. While it’s still important to include UI testing within your test automation environment, Cohn recommends choosing your automated UI tests strategically by asking questions like the ones we covered above.


Computer software case study

The challenges of Salesforce test automationTesting in Production: It's time