Continuous Testing with Hands-Free Automated Page Rendering Validation
You’re a developer, just finished the last release test yesterday and deployed to production. When you wake up the next day, there are 12 unanswered calls, 3 voicemails, and about 50 emails. Something about the page while in production was completely messed up and unreadable.
When you take a look, that’s actually the case, the entire site is completely messed up. You instantly panic.
Figure 1: Example YouTube page without CSS
OK, breath, let’s think. Why did this happen? You believed to have tested everything. This makes no sense.
Wait, you forgot to deploy the CSS. Let’s do that quickly.
Could this have been avoided? The answer is YES! It is very easy to avoid this situation with simple page rendering validation checks, that can be added to any functional test.
Does visual validation help teams execute?
Yes, definitely. Imagine an engine that conducts a smart (more on that later) continuous full page validation for every functional test flow, behind the scenes. Why is this important?
- Agility and developer productivity: Developers do not want to spend more time testing. Team leads want developers to develop, but they also want to avoid disasters. Executing visual validation on top of functional tests tells developers within minutes of code committed errors. Developers don’t need to remember what they did in order to undo a lot of their code. This is because the defect was found within minutes.
- Efficient cross-screen and cross-browser rendering validation: Like it or not, the page has to look good on all screens. Instead of repeating test flows for rendering validation, you can efficiently validate rendering in the background on all screens. Great efficiencies can be achieved because many of those experiences leverage the same HTML5 code base.
- Complementary to functional testing: In a typical functional test flow, only a few elements are actually interacted with. For example, in the login page, the flow would interact with the username and password text fields and then click the login button. A full-page visual validation would scan the entire page in one shot. Although it’s not a functional scan of each element, it is at least validates that the key elements are still there (or not).
Visual validation solution considerations
How would you differentiate between those? There are many aspects to this, I will describe the top considerations:
- CI integration: Are you able to gain insight into visual changes (suggesting potential defect) quickly, with scaling CI builds? Is it possible to integrate to common CI tools such as collaboration tools (ex: Slack), defect management (Jira) etc.? Here, for example, Applitools has a nice Jira integration to report a defect. The tool also offers the ability to comment on failures, encouraging developers to interact inside the tool.
- Different ways to create screenshots: Element screenshot, viewport screenshot, and full-page screenshot (viewport+ scrolling). This is very useful since an element screenshot can be used to isolate the comparison (from the rest of the page), reducing noise. Another benefit is when the page is built of components and you want to modularize your test. A full-page screenshot would be more appropriate for the traditional full-page rendering validation (for designers and marketers).
- How is the image analysis managed? This answer comes in two aspects:
- The algorithm itself: the scan method, and what tolerance threshold methods are available, ignore colors, antialiasing etc.
- The scan policy: this is one of the most important points. In many web pages there will be rotating content, content will change on a daily or hourly basis, perhaps advertising etc. The image scanning policy should allow defining “areas” where you may want to validate on a pixel-perfect policy. Other areas, for example, those presenting ads with rapidly changing content, you may want to block out. A good tool would allow you to block out multiple regions from a screenshot.
Figure 2: Example full page comparison result. Note the difference in Google search timing is highlighted
- Clear view and actionable dashboard/working environment: Is the reporting built in such way that it’s clear to quickly determine the issue when there’s a lot of tests running? Also, can you quickly mark a baseline, can you define a policy for an area etc.? Take a look at Appitools’ dashboard/work area to get a reference.
- Pricing model/support: Of course, there’s the issue of cost. Some tools are paid, free-to-pay, and open source. For example, Applitools has a freemium option, and then a paid version. Galen and others are open source, but this means that you cannot enjoy a premium support service from a vendor. The community around the open source solution might do a nice job, depending on your urgency, but it’s not guaranteed. Open source solutions also require ongoing hosting cost, not to mention building the assets that are needed, such as dashboards, integrations etc.
Embed visual validations into your process
Hopefully, this brief view of visual validation tools offers some insight into what can be achieved in a fairly low effort. I highly recommend trying it out on one of your next projects: Integrate the tool of choice, establish a baseline for the pages you want to scan, and from there on, let it run in the background. I believe you will find the defects found in this approach will not only offer you a new layer of coverage for “almost free”, but it would also make the marketing folks super happy.