QA Metrics: The Good, Bad, and Continuous
When trying to identify the goal of DevOps, it always starts by releasing new software fast, with clear value to customers, while reducing as much risk as possible associated with the new software pieces. Yes, that’s a clear objective to focus on but it’s harder said than done.
The digital transformation introduces so many risks that are associated with things varying from tools, people skill set, technology, and the target platform on which the new software will eventually run (mobile devices, desktop browsers, IOT, etc.).
With such complexities that are increasing, organizations ought to be thinking about key pillars to help get them to a “DevOps safe zone”, where most risks are known, mitigated, and software releases meet their objectives both from a user-story perspective, quality, and velocity.
Key Pillars for Continuous Testing in DevOps
Researches have shown that organizations that embrace the following 4 pillars are better positioned for success in their journey to DevOps.
- Having the ability to create test automation using tools that match the skill set within the organization is key.
- The ability to execute the created test scenarios against the right and required target platforms on-demand. All this from within the creative environment is crucial to gain quality visibility and address issues pre-production.
- When the target platforms are not provided as a service, and the lab isn’t stable, outdated, and insecure, the entire dependencies from both Dev and Test fail.
- Lastly, executing at scale continuously generates a pile of test data. Not having the ability to assess risks fast, remove noise, and focus on issues fast will slow down all of the DevOps pipeline activities.
Make sure you have these key pillars for success as part of your software delivery solution, shown visually below.
You Can’t Control What You Can’t Measure
Having the key pillars such as lab, creation, execution, and analysis is great, and imperative for your success, however, how can you assess whether you’re heading in the right direction? It all comes down to measurements and KPIs.
Setting up measurable goals across the various silos in the organization is very important. Setting this up will allow not just assessing immediate risks to software releases, but also, set the stage for continuous improvements in the future.
Here are few metrics that DevOps teams who are tasked with continuous testing and software delivery can embrace and measure (obviously there can be more metrics and some industry-specific metrics the team can and should add to the list):
- Development productivity – # of user stories delivered per software iteration?
- In such a metric, there needs to be also some trending visibility associated to show growth, static phase, and a decline in the metric
- Root Cause Analysis (RCA) –
- Average time to resolve defects
- Main reasons for defects to occur (software issue, platform specific, test environment, etc.)
- The pipeline, CI, and Build health – # of broken builds, length of CI trends (Up/Down/Solid)
- Continuous Testing
- % of test automation
- Pass/Fail ratio per software area
- # of defects detected within the build cycle vs. production defects
- Time to analyze software quality
- Test environment setup and stability as RCA for issues
- Test result consistency % (flaky vs. high-trust test automation)
As mentioned, the above is not the complete set of metrics DevOps team should care about. Although these are some significant ones when you sum them up, they are all connected to both the development and testing aspects of DevOps. DevOps is all about releasing with confidence, mitigating risks and providing value to the customers. It is a joint objective for all counterparts, hence, measuring the productivity, efficiency and reliability of all the activities that go into the process are the responsibility of the entire team. The business at the end of the day either gets a high-value release or not.