In Blog

machine learning

“How Machine Learning Is Being Used in the New Age of Test Automation Tools” – Webinar hosted by Software Test Professionals and presented by Dror Todress

Here’s a 5 min. recap of machine learning and its use for test automation webinar we presented in Software Testing Professionals (STP).

Our CEO and Co-founder, Dror Todress, had the privilege of discussing, “How Machine Learning Is Being Used In the New Age of QA Automation” in a webinar with STP. We are all quite aware how hot the topics of machine learning and artificial intelligence are in the tech industry. Often, QA managers and manual testers are interested in the topic but seem to fall behind on these trends and what they actually mean. Thus, Dror spoke thoroughly on the ML functionality in QA processes, with day-to-day testing examples and statistics. Dror ended the talk with a Q&A session (added below). You can watch the recorded webinar here.

Machine learning in digital transformation

Dror started off the webinar with the ins and outs of the digital transformation we are now facing. The shift of software development from waterfall to Agile forces your release cycles to shorten and keeps your updates constant. This means there is a constant consideration of ongoing maintenance of the test scripts. For this to work properly there needs to be an automation testing framework in place. Unfortunately, apps continuously change, and the tests need to be maintained and updated by the QA team. In this case, automation and ML are critical for frequent releases and to assist the team during the development and release cycle. With test automation and ML capabilities your testing team will be able to overcome those changes or easily adapt the tests to fit the changem catch defects before release, have more time for innovation, and more.

The importance of ML with codeless Selenium

As mentioned earlier, machine learning and automation are critical to the Agile method. Dror drilled down our ML methods and how our algorithm is working to be able to automatically overcome changes in the app and keep tests from failing. TestCraft Codeless Automation can overcome 97.4% of the changed in the app (changes that were not caused by a change in the app logic or process).

machine learning algorithm

Q&A

Dror closed the webinar with a fascinating Q&A session with the viewers. The questions mostly consist of ML questions and we provided them here for your benefit!

Q: How do we manage sites that are more intricate with high complexities that drive changes using machine learning and codeless Selenium?

A: TestCraft is fit for both simple and complex websites and we automatically apply the ML algorithm to every test step that is added to a test scenario. The testers will first need to define the test steps, using our simple visual methods of doing so. We will then translate those steps into Selenium-based code and apply our ML technology. It will apply if you have a simple process or, like some of our banking and insurance customers, a sophisticated one.

Q: Can you show us an example of a codeless Selenium test relying on dynamic data?

A: We would be happy to show a detailed demo where we can take you through actual examples of our platform in action. Please refer here to sign up for a demo.

Q: Are there any other ML usage areas in test automation?

A: Yes, there are. They could be in different parts of a testing cycle like analysis and helping users decide which tests to run when. Although, our focus today is on maintenance – our ML method automatically overcomes changes in the app and keeps tests from failing. This is also used to improve the test coverage.

Q: How do you collect the learning data sets? How is the model trained and what data is used for training and validation?

A: The data is being collected automatically by our platform. All the tester has to do is point on the screen, specifically, on an element. This way we can collect all the info and attributes of those elements. We do this for every element of the test and continue to collect the data and learn in every test execution. By doing this we are able to improve our results.

Q: You’ve shown many metrics and calculations that are taken into account as part of the ML algorithm – How does it affect runtime – does it increase the test execution time significantly?

A: It doesn’t slow down test runs.

Q: Is codeless Selenium available already? If yes, how?

A: Yes, it’s been available for the past 3 years with our platform and widely tested and used by our customers. Feel free to explore the solution further.

Q: How many selectors do we need to use as the input to achieve the level of accuracy? Or does the system locate the items themselves?

A: You don’t need to set up anything with our platform. The system collects all attributes for each test element, although we do allow the more advanced users to intervene. They can do this by identifying the weights for specific elements in an attribute. This helps us improve the results further. For example – a login button on the screen, you can safely assume the text will not change anytime soon. So then, you can direct the system to give a high weight to the inner text attribute of that element, being ‘login’. Because of that the login button can change its color id or location on the screen while  still saying ‘login’ and expect it to be used for a login action. This improves the resilience of this step.

Q: There can be DOM Trees with thousand of elements and perform a lot of calculations to find an element. How do you work through this?

A: We do it efficiently by benchmarking 30K elements with less than 200ms of working on all the DOM. There’s NO significant delay.

Q: Please talk about the actual workflow involved in creating and running the test while using your platform, a day in the life example.

A: As soon as you log into our platform you have an online canvas and can set it for a specific test and open up the basic file (canvas) to create a flowchart. That flowchart consists of different elements and actions that represent elements on the application you are testing. By connecting them in a certain sequence, you create your flow. Now, there are different ways to create a flow: 1. Drag and drop elements on the canvas and 2. Launch a session on our platform and the tested app and go through the process you want to create (point and click), so you can actually click and assign an action on it (validation, double click, etc). Once you do that, the flowchart’s code is being written for you in the background.

Q: How easy is it to add the ML algorithm to existing test cases, for example in UFT and regression tests. I want to automate the login page, how can I use the ML algorithm?

A: Adding ML to other platforms or to existing tests I believe is not an easy task, but I can’t’ really speak to how UFT might do that. It’s more for the modern solutions that are built specifically for Agile testing, but while using our platform you don’t need to do anything special for getting the ML algorithm into action. When you start to automate the login process you simply navigate to the app and perform the login process on the screen. (This saves as a data set). Once you click submit or login it will save and you have a ML login in place. It only takes a couple of minutes to automate a simple process.

Q: Is there an algorithm to identify image patterns? In some cases, we can’t interact directly with elements and we need to use image recognition (OpenCV for instance).

A: I believe there are different tools for visual testing but it’s not an area of focus for us. We do have an integration with Applitools if you want to incorporate a visual check within a test.

Q: How does this work with Ajax and elements that appear and disappear on the screen?

A: This happens to be an area of focus for us. First of all, before we run a test on a page we wait for the page to fully load and then we wait for Ajax codes to complete before we run the test. We allow a user to add different wait times or wait for a specific element to load giving different thresholds or different timeouts for those specific elements.

Q: Can we use ML to analyze a defect in the platform and make our automation more efficient?

A: Not with our platform at the moment. We do give you tools for doing that so you can analyze different executions for test suites over time. The tools allow you to compare and identify patterns leading you to improve the quality of testing.

Q: Is the code generated on your platform readable, exportable, and in a specific language?

A: Yes, yes. and yes. One of the advantages for us of using Selenium is we are able to provide customers an ability to export their code that is being generated automatically on our platform. The code is in Javascript, although it can’t be edited and uploaded back in again, it is more for continuity purposes. The code is your asset if you created it and you have the ability to store it and use it for another platform if you need to. Many solutions that use ML and don’t use Selenium for their underlying code won’t allow you to export and won’t allow use on other frameworks.

Q: Does your ML algorithm cater for automating performance and security testing scenarios?

A: Not on our tool. Our focus is on end-to-end functional testing, not load testing at the moment.

Q: Is there a benefit of using your tool for a Legacy app where element selectors do not change?

A: Definitely yes. The test creation process is much easier with the use of codeless automation no matter what.

Q: How do you search for an element that doesn’t have attributes?

A: We work on the DOM tree, so if the elements have attributes on them, they would become less resilient.

Q: Can I couple it with Cucumber? 

A: Not at the moment.

Q: Currently my company is using Selenium for automating web applications, if we want to move to ML what do I need to know? How can I use ML for my existing automation projects?

A: You will probably need to go through a long process of hiring engineers that specialize in machine learning and have them write the code. It is not a simple project or one that I can simply give the instructions for.

Q: Are there any features in Selenium, i.e. any testing that can be done with Selenium but is limited for TestCraft?

A: Not that we’re aware of. We have recently launched “Selenium Bricks” which enables to take any Selenium library, wrap it up, and include as a test step, in case you come across a capability that we haven’t yet developed.


Selenium Testing eBook