In Blog

continuous testing

Enable Continuous Testing Through Test-Driven Product Development

The market is very much aligned with the need to shift to Agile and DevOps. They understand the importance of automating the tests throughout the entire DevOps pipeline, however – who owns quality in a DevOps reality?

In this article, I’ll explain that regardless of the software development method, teams need to build software with quality and test automation that works.


Agile testing can be implemented in various ways:

  1.    BDD – behavior-driven development, where business, test, and dev collaborate and align on the features and scenarios to implement. This while using a straightforward scripting method.
  2.    TDD/ATDD – this is where testing occurs prior to features implementation to remove quality risk as early as possible. In such case only when the implemented code passes the pre-developed tests, the feature is good to go.
  3.    COE and cross-team testing services unit –given that, there is an advanced test automation team that owns an automation framework that is used by the various feature teams.

True, there might be more approaches, but our main focus is not the method to sustain agile testing. Rather, learning the approach to having the ability as a cross-functional team to continuously develop robust test code from the early design phase and user stories creation throughout the product release.

As the subject of this article suggests, every product in an agile practice ought to be developed with high quality in mind.

Quality Within the DevOps Pipeline

In an agile reality, software is being released either upon a code commit or in very short iterations of 2-3 weeks. To enable continuous quality per each code delivery, the scoping of tests that are being developed, executed and maintained need to be very lean and relevant to the product changes.

To meet the above objective the testing function in the organization needs to be involved as early as possible in the design phase, user stories development, and architectural discussion. Only by being involved early and throughout the process, QA can influence and drive the product quality forward in an efficient way.

Typically, teams should think of the following pillars from a scoping perspective:

✓  Unit testing

✓  Functional testing

✓  Relevant regression tests

✓  Non-functional suite

All the above-mentioned test types should be as much as possible automated and be plugged into the continuous integration workflow. The size of tests, the coverage of platforms should be a very well thought process that is based on the feature and code changes implications, the target persona for the feature, and some known analytics that can guide on past usage patterns.

You can understand that when the QA management is involved in all the above, there will be a match between the desired outcome and the end results.

From a test authoring standpoint, development and maintenance of the suite above are a hard and ongoing responsibility for the test owners, in many cases can be developers, testers, and performance engineers. To ease the maintenance, test code should be treated as production code, hence, managed in a source control management system (SCM), where history can be tracked, defects, and versions of the test code can be consumed. 

Responsibility for Achieving Continuous Testing

There is no doubt even in the most advanced agile organizations, that the quality function owns responsibility for product quality, release criteria, and release decision making.

As described in this blog, there are many moving parts continuously in a product life cycle. These moving parts involve defect management, test design activities, test execution in parallel within CI, and test report analysis. Dev teams are focused on building what’s right, taking care of coding these innovative features, and solving their bugs – however, they always need an objectiveeye” that oversees the entire product quality, risks, and end-user requirements that come from the business and more.

The Role of Quality in an Agile/DevOps Practice

The testing function should drive the entire quality agenda that includes:

✓  What to Test?

o   Test scoping is key for success, based on code changes evaluation test engineers shall define the scope of a test cycle

✓  What to Automate?

o   Not every test deserves to be automated and re-used on an ongoing basis, quality is best positioned to take decisions around that debate.

✓  Which Platforms to Test On?

o   Leverage analytics, regression history results, customer insights and market events to guide your test coverage decisions

✓  What is The Quality Release Criteria?

o   Setting the release criteria threshold is one of the most important aspects, hence needs a strong quality background to set it up, fine tune it as a product evolves, and enforce it.

✓  Test Maintenance and Test Lifecycle

o   Each test has its unique lifecycle, and either requires ongoing changes, or retirement when it is obsolete, duplicated etc. Quality should stand behind these maintenance decisions making at all time to assure test suites are up to date, cover sufficiently the quality criteria and do not cause redundant overhead.

Bottom Line

We hear a lot of new and innovative methods that are introduced to the market that offer machine learning (ML), and artificial intelligence (AI) capabilities to the agile teams. These are on one hand only automated tools that evolve, and on the other hand – they need a “quality controller” that can steer them towards the right decision making so there are no escape defects, quality coverage holes, and regression in the overall Agile/DevOps organizational journeys.


Selenium Testing eBook