In Blog

The Biggest Change You Might Have Missed in Gartner's Magic Quadrant Report


For many years Gartner’s Magic Quadrant Report has been one of the main sources managers read in order to decide on their QA strategy. As always, we find their most recent report interesting and decided to cover it and summarize it for you.

At first glance at the Gartner’s report, we were presented with a great list of strengths and weaknesses for the top software test automation companies in the industry at present. Gartner made in-depth analyses on multiple companies that they laid out as either the leaders, visionaries, challengers, or niche players of the current industry.

A Major Change from Last Year’s Gartner’s Magic Quadrant report

On further inspection, we noticed something is different than last year’s report – after they didn’t mention Selenium last year, and were criticised for it, this year we can see that their perspective had changed and Selenium is mentioned in various different instances along the report. 

Joe Colantonio, the founder of Test Talks, noticed the missing link last year. He stated, in his article Magic Quadrant for Test Automation: How Gartner got it wrong, that, “it was a mistake for Gartner to leave Selenium out of its report as a viable option for software test automation. I understand that it didn’t meet the criteria Gartner clearly listed, but to my (and many others’) way of thinking, it was a glaring omission.”

It is clear that Selenium use and its community is growing and can’t be ignored. Just take a look at Google trends stats comparing Selenium to some of its “competition”.google trends

Open-Source Solutions

Although we believe that Selenium should have been even more prominent in Gartner’s report, we still feel there were a lot of important highlights in it. The strengths and weaknesses of the companies that were listed can help any manager be able to pick the right test automation framework for them. Your task now is to look at the main strengths and weaknesses below and choose which strengths are most important to you and which weaknesses you are able to live with.

Strengths of Test Automation Solutions

It’s time to recognize the strengths of a software test automation service according to the Gartner report. Every service has a different strength and weakness, but the goal here is to find out what strength is important to your company’s needs.

  1. Open source – to us the most important strength. Some companies believe that having their own environment, such as IBM, have it all figured out. Honestly, it is more important to be a part of the Selenium ecosystem, that way you, as a tester, aren’t limited to the tools you can use.
  2. It is important to find a tool that offers a shift left testing approach through a Saas-based platform, where it allows open source software that supports continuous testing.
  3. Allows the tester to manage test cases and data changes during the test runs
  4. Imperative to find a service that has multiple integrations. You can’t expect one service to have it all on its own mainframe.
  5. A few big test automation companies have a broad range of partners that will be able to extend a helping hand to their customers
  6. Behavior-driven testing, which will increase agility and productivity within your company.
  7. Easy to use framework – one that anyone can use with the right training. One that is not only for developers

Weaknesses of Test Automation Solutions

When it comes to weaknesses it is important to mention that every service has a couple. You will never be able to find the perfect tool, but this will help you find the most suitable one.

  1. There is a weakness when it doesn’t offer support for UI. If they offer this by using an integration, you might be forced to pay an additional cost for the separate tool used.
  2. Some tools might not have the most updated toolkit and it’s important to have the most modern toolkit to use to your advantage. There is no reason to be behind in your automated testing.
  3. If a portfolio or framework is too complex to use. You might want to find one that has a simple user interface so anyone, not only developers, can use the tool.
  4. It is possible to have too many integration and partnerships. There can be bad overlap between the integrations.
  5. If the framework lacks record capabilities.
  6. The service didn’t offer the right amount of training to use their tool properly.

Hopefully, after looking at all the strengths and weaknesses you, as a manager or anyone that has influence in their company, can choose correctly what is best for them. The main point to remember is the importance of the Selenium ecosystem. Why stick to an environment that only allows specific tools? Selenium is an open source framework that allows for multiple tools to be used within the ecosystem. Don’t limit your testers, let them excel with an open source solution.   


start Free Trial