Two Approaches, One Process: Functional and Visual Regression Testing

By

Hugo Farinha

,

Co-founder and CTO

2021-02-17

Functional testing and visual regression testing are two distinct testing activities. Sure they both look at a UI but they have different approaches, different objectives and different toolsets. But are there advantages to using the same toolchain for both while respecting the fact that they require completely different approaches? Well, that is what this blog post is going to try to find out. Let’s start by defining exactly what we mean by functional and visual regression testing. 

  • Functional testing validates that the application under test meets functional requirements/specifications. To put it simply does the application work as it should. 
  • Visual regression testing validates the look and feel of the user interface against the requirements/specifications.

From these definitions, you can see why they both require a completely different approach. But both do answer the same fundamental questions: is my application working as expected and ready for release? 

Although these two types of testing have been opened up to test automation they have traditionally been done by manual testers checking that both the functionality and look and feel of an application is as expected.  When we view these processes from the prism of manual testing the two become less mutually exclusive and we can see an overlap. A manual tester wouldn’t ignore a layout issue whilst testing for functionality or vice versa. But the disadvantages of manual testing are well known: 

  • Test creation and execution is slow
  • Test coverage is reduced as it is not scalable.
  • Is prone to human errors especially in the UI layer 
  • Repetitive unimaginative tasks. 
  • Defects are found later in the SDLC phases at a higher cost, The Systems Sciences Institute at IBM reported that it costs 15x more to fix a bug found during testing phase than to fix one identified during design.
  • Slower release cycles
  • Higher risk of defects not being found until the software is in production

To address those issues teams automate tests at all levels including functionality and visual regression using scripts. At the UI level, the irony is that only functionality is tested by the automation tool in most cases. This is not a real visual regression testing, which introduces its own problems:

  • Still takes a long time to create and maintain the UI scripts
  • Try to simulate UI validation but no real visual regression testing is done at all
  • A working application is needed to be able to automate against, which tends to mean you manually test first and then automate when it is working – a very slow approach to testing.

When automating testing in this way you merely simulate visual regression testing as scripting rules for visual regression testing can only validate applications at the DOM level. You cannot guarantee that the UI meets the visual specifications. Validating an element’s position is not the same as ensuring that the element is shown according to the specifications. Validating that an image has the correct given URL and is loading successfully does not mean that the image is the right one as the designer may have saved the wrong image, etc.

Snapshot testing

The snapshot approach to automation is very different from the one used for functional testing. Is it then wise to use the same tools and processes for both? As we have seen manual testers cover functional and visual regression tests page by page and state by state. A path is followed through the application which ends at a certain page or state of the application. Functional tests typically follow these same journeys. Therefore, to combine functional testing with visual regression testing we just need integrations with tools that offer support or the right tool that supports both functional testing and layout testing. Automation wise, the effort for the layout testing comes almost for free as we are executing snapshot testing based upon journeys created by functional testing. By using the same tool for functional and visual regression testing you not only drastically reduce the time it takes to create and maintain both you also half the costs of executing both processes separately.

Several tools have attempted to offer a solution which was a great step forward. These tools use Selenium or other similar frameworks to drive the functionality as normal, but append into the asserts a UI check. This removes the need to have two sets of automation tests, but still leaves you with:

  • Tests that are slow to create and maintain (although faster and better than pure selenium approach to do simulated visual regression testing)
  • You need a working application to be able to automate against, which tends to mean a manual test first need to be available and then automate when it is working – a very slow approach to testing

Virtuoso picks up the baton from here and lets us leap over these remaining two hurdles. And when you consider the speed at which DevOps are expected to deliver, and therefore test, overcoming these hurdles is critical. The speed at which tests can be authored using Virtuoso is comparable with authoring a manual test script. The script can also be created much earlier before the application is developed. Put simply, if you can write a manual test script, you can now write an automated script that covers both functional and visual regression testing processes. How this fits into your CI/CD pipeline is explained in this shift-left blog post.

With Virtuoso the maintenance is also reduced as only one script is needed. But Virtuoso goes one step further to reduce maintenance, Virtuoso self-healing technology reduces the maintenance to as close to zero as you can get.

What is the advantage of separating functional and layout into different tools and processes?

By having the same process it doesn’t mean there can’t be two different reports coming out of a single execution. It is, in fact, important that these two reports are distinct. You want a report for functional testing and a report for layout testing, as you don’t want to fail every test because there is a minor UI issue. So having two reports in the same process, not only saves time, effort, and money it also allows you to:

  • Have different teams to validate results
  • Clear reasons for failures
  • Keep functional and UI test fails separate
  • Focus on what has changed, and prioritise testing

The disadvantages of not combining the two are increases in: 

  • The amount of work
  • The testing time
  • The cost to execute

Conclusion

There are clear reasons why functional and visual regression testing requires a different approach. However, this does not mean that the test automation tools and processes used should not be combined. Reusability of automation scripts for functional and layout testing is key to speeding up the testing process, reducing test maintenance, and execution costs. While having functional and layout separated at the reporting level you can easily distribute work across a team, clearly identify reasons for failure, and keep functional and visual regression test fails separate. In doing so, we can create the right conditions to have the answers that give us the confidence to release, release often, and release faster. 


Tags

Building Test Automation
for True Shift Left

How to shift your testing left and build web automation that keeps up with your development.

Download eBook

Subscribe to our Newsletter

@ Copyright 2021 SpotQA, Creators of Virtuoso – PRIVACY POLICY