Automated test creation, which approach is right for you?
When we compare the different approaches to test creation, can we really say that one approach is definitely better than the rest?
In this short post we provide a quick overview to three very different approaches to automated test creation; Record and Replay, Scripting and NLP, their pros and cons and ask ourselves “Is there a better way?”
Record and Replay
Let’s start with Record and Replay. This approach allows the user to easily follow a natural journey through the application without ever having to think that they are actually "testing" it. The user doesn't need to know what commands to use and it requires little or no training to start with automation. While recording, the user’s full attention is focused on achieving the journey rather than testing/automation needs.
Record and Replay is more precise with element selection. It can automatically capture a wide spectrum of deterministic selectors that will exactly match the element the user interacts with. But, it’s also a more “organic” approach to test authoring because it’s much closer to how a genuine user would likely behave. With Record and Play testers can simply interact with the application as if they were a real customer.
In testing, there will always be scenarios that are so exceptional and specific that codeless interfaces will not be able to address them. In these cases, you would be able to rely on the power of scripting.
Being able to script means you are able to cover every single angle and detail inside your application scope, as well as interacting with external systems. Whether it’s preparing the test environment conditions or validating that a third party received and processed an event triggered by the application, the number of practical applications for scripting are innumerable.
The way scripting can be designed as a library for reusability, and its content easily edited, make it even more flexible and versatile for your needs. The control and accuracy that a user has over interactions with the application, environment, and external systems is probably the strongest advantage of scripting.
However, the downside of scripting is simply the cost. Development cost is significant because the user needs to know how to code. He/she also needs to understand the basic principles of software engineering to be able to address different systems and environments.
The maintenance cost is also high because when the application changes the scripts are often no longer valid, fail to execute, and require fixing. Because of these disadvantages, the use of scripting should be as a last resort for when everything else fails.
Natural Language Programming (NLP)
Since NLP tests are deliberately written or converted into plain English they are more likely to be readable. Also, since users have already seen the tests once, they'll recognise them more easily and require less time to inspect then in the future. Studies in software testing support this (see a paper published by our own Head of Engineering, Sina Shamshiri https://ieeexplore.ieee.org/document/8367053)
Using NLP also gives users the ability to write automation regardless of the current application version having been deployed or not. This gives users the power to effectively shift left and start building automation earlier in the development phase because the tests do not depend on an exact version of the application.
Similar to scripting tools, NLP has increased reusability, control and accuracy over the automatic recorders. These often have to guess user interactions and suffer when recording complex interactions due to corresponding technical challenges. With NLP the user has full control and can describe exact user interactions.
Maintenance and augmentation of tests created using Record and Replay can be cumbersome, often requiring a replay of the journey. Meanwhile, in most cases with NLP, users can make the changes directly to their tests.
Let’s not forget about portability, too. You can use a wide spectrum of technologies to write tests, from your smartphone or tablet to a simple piece of paper. For privacy-conscious users, with NLP there is no risk involved with installing an extension which can record everything they do and write.
Moreover, there is no dependency on a software application needing to be installed on your machine or browser for you to be able to write automation in plain English.
At SpotQA, we ultimately believe the closer we can bring the authoring tools together, the better we can bring teams together and provide a richer user experience. Therefore, within Virtuoso, we support and offer all three approaches, seamlessly integrated with one another.
Here are some examples of how a hybrid approach could feature as part of a test automation strategy.
- While recording using a Record and Replay, the user sees their journey on the embedded journey view on the application. They can manually write/change steps as they build the journey inside the application under test.
- If a user partially writes a journey in NLP and runs it, the journey will continue recording from there using the Record and Replay.
- While creating a functional journey that performs critical backend actions on one or more systems, the user may write a script library to make API calls to validate changes and data accuracy on backend and external systems.
- Similarly, a user would be able to run a journey originally created in natural language on their extension, step through them with breakpoints, and make changes live when maintaining the tests. This can be really useful for when a journey breaks and the user has to first get to the target page before they can provide selector fixes etc.
In short, there is no one-size-fits-all approach to testing. The varied demands placed upon QA by dynamic applications and CI pipelines call for a mix of techniques. The reality is that some applications are more complex to test, they integrate with more systems as part of complex business processes. Scripting will be required in part. But for the lion's share of testing why script and maintain code when you can take advantage of speedier, more accessible ways of testing? It all comes down to need and preference.
What sets Virtuoso apart from competitors is the fact that everything is combined in one platform. Whether you record it, script it, describe it in plain English, or let bots generate the tests for you, Virtuoso takes your testing further with lightning-fast execution, maintenance, and autonomous healing of your tests.