How ten24 reduced their bimonthly regression testing time by an average of 73% and helped their QA scale

 In Case Study

After eight months of working with TestCraft:

Lines of Selenium code

2K+

Test runs executed

Number of automated tests

~$400K

Saved in test automation costs

1700+ 

Hours of test creation and execution saved

Highlight:

ten24, a software company focused on building comprehensive eCommerce solutions for their clients, wanted to reduce their project turnaround time for their various clients. To do this, ten24 determined that they needed to shorten their software release cycles by reducing their testing bottleneck that often stemmed from regression testing. They needed a user-friendly, automated testing tool that could handle end-to-end testing for Angular applications with sizable JavaScript libraries. This tool would also have to allow their QA team to speed up and scale their testing efforts without requiring any previous coding skills. These needs led ten24 to start using TestCraft as their codeless functional test automation tool.

Results:

  • ten24 reduced their regression testing time from an average of 3-5 days to as little as 1-2 days during their 2-4 week sprints.  
  • ten24 minimized their technical debt significantly, due to a combination of TestCraft’s AI-based self-healing mechanism and its ability to catch bugs quickly.
  • Due to TestCraft’s simple and intuitive UI, ten24 was able to catch holes in their tests that they wouldn’t have identified previously. This allowed them to create more thorough tests when creating future projects.
  • TestCraft has taken our efficiency and overall product quality to a whole new level. We would have never tested as much as we do now as quickly as we do now without it.

    Kevin Jason Director of Development at ten24

What ten24 was looking for in a QA automation tool

For many eCommerce companies, choosing the right platform to operate their business can often feel like a daunting task. To address this, ten24 created Slatwall, a comprehensive eCommerce software platform, so companies can run their eCommerce businesses more easily and in a more streamlined manner. Customers ranging from Sotheby’s to Scientific American Magazine use Slatwall to respond to new changes in the eCommerce landscape without changing how they operate.

To reduce the time spent on delivering or releasing updates to their customers’ platforms, ten24 searched for a functional automated testing tool that would help them test more quickly and efficiently. With a small QA department that was only doing manual testing at the time, ten24 was looking to free up time and resources for their testers as well as speed up their regression testing and end-to-end testing processes. This would also allow the QA team to focus more closely on the manual tests that actually require human involvement.

When searching for a solution, ten24 opted for a codeless test automation tool after deciding that they needed the following from their automated testing platform:

  • Compatible with Angular. Many of ten24’s projects are based on Angular code, and they felt, when building their test automation tool shortlist, that many solutions could not handle their JavaScript library in the way that they needed.
  • User-friendly. The QA team at ten24 is comprised of business testers who do not have the technical background to work with a code-based test automation tool. Because of this, ten24 prioritized a tool with an intuitive UI that allowed for simpler test creation and execution.
  • Scalable. The ten24 team started their test automation journey with Selenium but found it tough to scale and maintain over time. They did not have the resources to bring in engineers to manage Selenium testing, nor did they feel that it would be the best way to allocate company resources. In order to scale, ten24 needed a long-term solution that would grow with the QA department over time.

A strategic and simple onboarding process

Since ten24 deploys eCommerce platforms for multiple companies, they started their test automation journey by focusing on select end-to-end testing and regression testing projects. They chose these projects as part of a larger strategy to start with tests that they felt would benefit the most from automation.

While any new tool requires a bit of a learning curve, ten24 found that TestCraft was very easy to learn and implement into their environment. TestCraft’s user-friendly platform made onboarding new members of the ten24 QA team both simple and efficient.

ten24 also felt secure in their ability to rely on the TestCraft team for support. They felt that the TestCraft team was always available to address questions about the platform, as well as offer best practices about the tools that could help them move forward more efficiently.

Reducing test creation and execution time with visual modeling

One of the first projects that ten24 chose to automate included a regression testing suite that they run every 2 weeks. After automating this suite with TestCraft, ten24 reduced their time spent on their bimonthly regression testing, from 3-5 days of manual regression testing to just 1-2 days. This not only saved them hours of regression testing time, but shortened the entire project by allowing the QA team to test manually with the automated tests running in the background. In other projects, ten24 was able to minimize their testing time even more, from half a day of testing to just fifteen minutes.

A graph showing how TestCraft helped reduce ten24’s regression testing time

A graph showing how TestCraft helped reduce ten24’s regression testing time

ten24 determined that they were able to build and execute their regression tests both quickly and efficiently due to TestCraft’s visual test modeling capabilities. They found TestCraft’s drag-and-drop feature to be particularly helpful because it made test creation a much simpler and more intuitive process. By creating a model (as opposed to a test recording), ten24 was free to reuse test steps easily for future test scenarios. Any changes made to these steps would then be applied to all other scenarios that used those test steps. This reduced test creation time as well as test maintenance time dramatically.

Example of using reusable test components in TestCraft

Example of using reusable test components in TestCraft

TestCraft’s drag-and-drop feature also created a positive ripple effect on future projects. By allowing them to create new test suites quickly and easily, ten24 recorded that they could create new test suites 1.5 times more quickly. For example, if it took ten24 five days to complete their first project from the initial strategy to full completion, it would take four days to complete the next project and three days to complete the third. By taking advantage of reusable test steps feature, ten24’s was able to benefit from long-term time savings that were sustainable.

Chart showing the effect of reusing test steps on ten24’s project completion time

Chart showing the effect of reusing test steps on ten24’s project completion time

  • Reusing test steps in TestCraft is great. It made our project turnaround time consistently faster.

    Jada Flournoy QA Lead at ten24

Eliminating human error when catching bugs

TestCraft’s scheduling feature and simple test creation capabilities gave ten24 the flexibility to run a wider range of tests more frequently, which enabled them to catch bugs more quickly and more easily. This allowed ten24 to find issues before their clients did and avoid these bugs later on in the regression testing cycle, saving them the future costs of resolving bugs at a later stage of development.

Using TestCraft also helped ten24 ensure that their tests were executed thoroughly, regardless of their frequency. They recognized that with the number of times they wanted to test their core end-to-end processes, manual testing would not cut it. Humans would either skip tests or make mistakes due to frequent repetition. With TestCraft, ten24 was able to eliminate the “human error” that comes with manual testing, and reassured ten24 that their tests would run the same way every time.

  • TestCraft completely took away the human error component of testing. No matter how many times we went through the test or scheduled it to run, we knew that we could rely on TestCraft’s results.

    Jada Flournoy QA Lead at ten24

Another way TestCraft helped eliminate human error when testing was by providing a different point of view about tests. Not only would TestCraft catch bugs earlier in the testing process, but the platform would also point out errors that the QA team wouldn’t necessarily consider or find on their own. This different perspective helped QA look at tests differently, which helped them later on when documenting how to build test suites for subsequent projects. 

Reduced technical debt and costs

The ability to catch bugs quickly also helped reduce ten24’s technical debt tremendously. Since ten24 built the vast majority of their tests using reusable components, the issues that they resolved often applied to multiple test suites. TestCraft’s AI-based self-healing mechanism also helped reduce technical debt by ensuring that ten24 was creating resilient test scenarios that could scale.

Ten24 was not only able to speed up their end-to-end testing and regression testing, but they did this without having to hire any additional personnel to conduct automated tests or focus on maintaining a code-based tool like Selenium. The ability to create and execute test suites quickly while maintaining their existing QA team made a major financial impact, saving ten24 roughly $400,000 in personnel costs annually. These cost savings allowed ten24 the flexibility to better allocate their resources for future company needs.

ten24’s TestCraft functional test automation dashboard after 8 months of using our codeless testing platform

ten24’s TestCraft functional test automation dashboard after 8 months of using our codeless testing platform

Codeless testing improves test automation strategy

While ten24 reaped the various test automation benefits that TestCraft had to offer, they also found that it had a positive impact on their overall testing strategy. From the beginning of their test automation journey, ten24 had a very clear plan about which tests they wanted to automate and implement. Yet they also understood that while you can have a great plan, the plan can change when going through the actual tests themselves and running through a wide range of different scenarios.

ten24 found that TestCraft helped bridge the gap between their original testing plans and what actually happened in real-time. As a codeless test automation platform, TestCraft made it very easy for ten24’s QA team to find holes in their tests. This helped them regroup and build future scenarios that would account for these previously overlooked areas.

  • You can have a great plan when it comes to your automated testing, but things can change when you’re actually going through the process. TestCraft helps us bridge the gap.

    Kevin Jason Director of Development at ten24

Defining a comprehensive test automation strategy, coupled with the right tooling, creates a great foundation for achieving your testing goals. With TestCraft, ten24 took their testing to the next level, which gave them the time they needed to keep their clients on the cutting edge of eCommerce.  

Free Trial

Please register here and start your free trial