Strengthen the QA process through the CI/CD pipeline [webinar recap]
Strengthen the QA process through the CI/CD pipeline [webinar recap]
For many testers, the following two scenarios will probably feel familiar: realizing that a bug has shipped in multiple releases, and being told that a new feature only works on one machine. These are two situations, both equally cringe-worthy and frustrating, that testers deal with all too often.
Two weeks ago, we had the opportunity to discuss a great way to address these issues with Christina Thalayasingam, Senior Test Engineer at Medidata Solutions: a well-running CI/CD pipeline. In a guest webinar with TestCraft, Christina discussed how integrating tests into the CI/CD pipeline will lead to stronger QA processes and ultimately, a higher-quality product.
Here is a brief summary of what Christina discussed in the webinar below, and you can also watch the full recording here.
Continuous integration vs. continuous delivery vs. continuous deployment
Christina started the webinar by giving an overview of continuous integration, continuous delivery, and continuous deployment. While doing this, she highlighted a few key points that drove home how testing continuously throughout the CI/CD pipeline helps companies release higher quality products.
She first pointed out that continuous integration (CI) is based on integrations happening frequently. Anytime there is a change or the development team wants to release a new software feature, the rule of thumb is to do comprehensive regression testing, since this will help testers encounter fewer bugs later on. Therefore, if you want an integration to be considered viable, testers need to make sure that they can detect integration errors as soon as possible.
Christina then continued to say that continuous delivery (CD) is an extension of CI by ensuring that these releases happen both quickly and sustainably. In addition to automating tests, continuous delivery allows for automating release processes so that companies can deploy applications at any time. The more continuous testing that is done, the better the delivery is after each release.
Continuous deployment is also something that requires strategic and efficient continuous testing, especially since it’s a completely automated process (unlike continuous delivery, which still has some manual components). This would need to cover the full software testing process end to end, including aspects of testing that you might not even have considered, such as reporting.
Continuous testing = full-stack testing
After this explanation, Christina continued to emphasize that continuous testing is a critical part of creating a CI/CD pipeline that will work in a reliable manner. She explained that the main reason why continuous testing is important is that it enables continuous improvement.
Also referring to it as full-stack testing, continuous testing should cover every aspect of the product, including:
- Test management (especially for reporting and analytics)
- UI-level functional & E2E testing
- Service-level functional testing
- Code quality and coverage
- Performance engineering
- Security testing
- Unit testing & test-driven development (TDD)
Continuous testing advice
In order to have effective continuous testing, Christina offered a few pieces of advice for those who attended the webinar. The first thing she suggested was to set a continuous testing strategy from the get-go.
As part of this strategy, she noted it’s important to choose tools that will fit your purposes, as well as your requirements, current technology stack, and how your team handles workflows. Choosing these tools should be decided unanimously by the QA, development, and DevOps teams to ensure that all teams can use the tools and collaborate with one another.
She also suggested that testing teams choose their integrations wisely and keep their testing suites separate. When a team’s suites are organized properly, you have the freedom to just focus on the automated tests that correspond directly to the changes being made to the product. This helps when keeping suites separate, and it also allows teams to prioritize which tests they want to run at a given moment, especially when there are time constraints.
Another piece of advice that Christina gave in order to do effective continuous testing is to make sure to include reporting in order to maintain a high level of collaboration among team members. Reporting, Christina emphasized, gives very high visibility of issues and helps other teams trust your work. Especially in the case when developers often do not trust when QA finds issues, reporting helps with clearly explaining the problem and ultimately building that trust.
Why testing in the CI/CD pipeline is important
To show the importance of continuous testing in the CI/CD pipeline, Christina ran a few live demonstrations of adding tests to the CI/CD pipeline. She showcased a wide range of tests, as well as the different tools she used to integrate them properly.
She first demonstrated how to add Selenium-based functional tests to the CI/CD pipeline using TestCraft. One important feature she highlighted was TestCraft’s ability to record videos of test runs, which she mentioned can help testers identify the test for future use. By integrating with Jenkins and other popular CI tools, she showed that integrating tests into the CI/CD pipeline with TestCraft was a simple and smooth process.
Other demonstrations that she did included adding test management and defect management capabilities, security testing, and performance testing to the CI/CD pipeline. Using tools like Zephyr, JMeter, ZAP, and others, she built workflows that cover the entire QA process from end to end.
Following these demonstrations, she shared that in order to get started with this process, it’s important to make the people in your company believe in the change that continuous testing can provide. Not only does testing in the CI/CD pipeline help facilitate better communication, but it also allows for the earlier detection and prevention of defects, reduction of manual testing efforts, and overall time spent on debugging.
Questions about CI/CD
Christina then ended the webinar by answering questions from the audience. For your convenience, you can read the questions and her answers below:
Do you have different roles for the members of your team, or can anyone create or prepare the CI?
“We can have at least a single tester in each scrum team who is aware of how to prepare the CI so that testing can be added to that. It is best that all testers are given the basics of the CI/continuous testing configuration so that they would be able to troubleshoot or understand if something goes wrong. For the initial stages, you can have dedicated personnel to help you and then proceed when most of the team has understood and gotten more hands-on in conducting continuous testing.”
Nonfunctional testing should depend on the result of functional testing. Can we set things up to run that way in the CI tool?
“Yes, you should definitely pass functional tests in order to proceed to non-functional testing. There is no point in running non-functional tests when the application does not meet functional acceptance. You can have the non-functional tests run as soon as the functional testing meets its exit code in its step in the pipeline. Here is an example of how this can happen in Jenkins.”
How is the automation team different from the continuous integration testing team?
“There is no need to have different teams manage automation testing and continuous testing. Continuous testing is nothing but the next step of automated testing. If your automation tests are robust you integrate those tests into continuous integration, which would enable continuous testing. Therefore, the testing team should be able to eventually understand the process and configuration and work on integrating automation tests to the CI pipelines.”