My test is failing, what now: How root cause analysis can deliver better software, faster

By

Sina Shamshiri & Ryan Thornton

,

Development & Product Marketing

2021-05-26

We often lose sight of why we're testing. Testing for testing’s sake is an exercise in futility. At the most fundamental level testing is done to help with finding and fixing bugs for users. A failing test does not directly benefit a user, nor does an open bug report; a bug found and fixed before a user can find out about it is what users value! 

Building tests is only half the battle, testers devote a significant amount of time to maintaining and investigating tests, often requiring more resources than test authoring. When a test fails we cannot say we have found a bug without further inquiry, more often than not, a failing test indicates new functionality or behaviour in the application that is expected, and so the test needs to be maintained. We need to get to the root cause of the failure for the test to be meaningful. 

Testers and engineers are not a cheap and plentiful resource, nor is releasing software that has failed tests without understanding why a wise decision. Therefore, the root cause analysis of failures represents an important area where efficiency and accuracy can dramatically impact ROI and may even be business-critical. How can we make sure our failing tests are meaningful? 

User-centred tests

Tests are only as good as the problem that they fix. Automated tests may fail in situations where an end-user would have no problem interacting with an application. This could be due to: 

  • The test needs maintenance, because: the application has changed; or the test is invalid (it is the first execution of it, and likely the wrong element is being targeted)
  • The environment is in a bad state (flakiness)

If we take it back to first principles it is key to test to improve the user experience, after all, there is no point in allocating resources to testing if you do not improve the product from the users perspective. To do so you need to know why a test failed. You need root cause analysis. Root cause analysis is quite straightforward to define but not always that easy to perform. Put simply, it is the identification of what caused a test to fail. It is a process that seeks to find the origin of a problem and answer why the problem happened. The advantages of root cause analysis are many and well documented: finding bugs earlier gives a greater ROI; faster time to market; etc. The challenge is adopting the right approach and tooling to take advantage of its benefits quickly and easily. 

Root cause analysis and Virtuoso

Those familiar with Virtuoso will be well aware that we approach testing on a holistic level, we help our users ship better quality products faster. Therefore, it should come as no surprise that helping teams with root cause analysis is top of our agenda. This is why we developed our root cause analysis feature which gives you all the information you need to locate the genesis of problems at a glance: 

  • Analyse the execution history of the journey and the step, people involved, etc.
  • Review the recent activity on the journey from executions, to changes and comments left
  • Compare visually the current failing step with what it looked like when it passed last time
  • Comment and discuss the failure with your team
  • Review network traffic, console logs, and the page source (DOM) at the time of failure

Once you have identified the root cause (e.g., whether it was due to a regression, flaky environment, or another reason) you can provide a reason for the failure and get all the information presented to you instantly in a handy report. As Virtuoso tracks your execution history even the most granular of information is at your fingertips. Focus right the way down to the test steps level and see screenshots, network traffic details, and much more. You can find out more about the feature here. 

Conclusion 

Root cause analysis represents a great opportunity for teams to improve the quality of their product, reduce maintenance overhead, ship faster, increase customer satisfaction. Virtuoso is committed to providing its users with features that enable them to find the underlying reason for a problem and not just automate tests. And as we combine this approach with the latest developments in AI/ML our vision is to accelerate the benefits of root cause analysis offered as part of our vision for the future of testing.

Tags

Building Test Automation
for True Shift Left

How to shift your testing left and build web automation that keeps up with your development.

Download eBook

Subscribe to our Newsletter

@ Copyright 2021 SpotQA, Creators of Virtuoso – PRIVACY POLICY