In Blog


Last month during one of the busiest of Amazon’s annual Prime days, its web services crashed, causing an astounding $34 million loss per hour of being down. While for web users it may sound cool to meet Amazon’s employees’ dogs, this is no joke for Amazon, considering the magnitude of its losses.

defectsAverage cost of a software defect

It’s not new that software defects are much more expensive the later they are found in the development lifecycle, more critical when they are detected in production. The obvious reason to the rising cost is the overall effort required both by developers to identify the root cause of the defect, resolve it, and perform the amount of testing to ensure the defect is fixed, and ensure the fix did not introduce a new regression.

To illustrate the above impact here are some market standard time estimates of defect resolution based on the time it was identified in the sprint, together with the time it will leave developers to focus on product innovations. A detection of defects the same day they were introduced into the software build allows developers to spend 47% of their business day on innovation and feature development while executing around 25% automated testing scenarios. Increasing test automation coverage to 75% would leave 77% of the time for a developer to focus on innovation.

defectsThe math is simple: more automation, with more testing done daily, close to the code-commit translates into efficient defect detection, resolution, and more time to spend reducing the product backlog.

Continuous testing to the rescue

Continuous testing (CT) is the process of embedding various types of software testing (unit, functional, non-functional) as a fundamental ongoing practice throughout the entire development lifecycle. Having the ability to verify the quality of software after each code change in an automated way in the CI process holds great benefits for all DevOps practitioners.

In the context of web testing and escaped defects, the more testing that is being done per code-commit, the higher the effectivity of the entire DevOps cycle and software iteration is. Effective continuous testing also employs an overall fast feedback loop, such that can address issues as they are introduced into the build cycle. At the heart of CT is a robust test automation framework that is used upon every code changes, to execute the right tests against the right platforms in the test lab.


Putting it all together

To avoid issues like what Amazon faced, or even worse, DevOps teams are required to proactively build testing into their DevOps pipeline and automate and test these scenarios as much and as early as possible. There is a need to shift-left and to build the tests early on, as to stick on the CT path. 

Continuous testing as defined above should offer all DevOps personas (Dev, Test, Business) a common ground for the entire testing activities, both functional flows, as well as non-functional flows, such include load testing, accessibility, security, UX, and more. Websites crashes, availability, and other production issues occur both due to functional issues as well as non-functional, that’s why CT that leverages a robust test automation framework is the key for success, since it enables the teams to build a suite of all relevant regression tests, and execute them pre-production, as well as in production as part of ongoing scheduled monitoring testing.

Finally, CT and proactive testing is a great thing for DevOps teams and should be employed at all times, however, for special market events such as the AWS Prime Day, and similar, the DevOps teams should most likely add an additional layer of validations with a larger significant load and availability testing, to identify robustness issues, 3rd party services functionality issues and more in advance and assures fluent serviceability to the end users’.

E-commerce case study