Record/Replay vs Scripted vs NLP Test Authoring

Published on
August 11, 2022
Hugo Farinha
CTO & Co-Founder

When we compare the different approaches to test creation, can we really say that one approach is definitely better than the rest?

In this post we provide a quick overview of three very different approaches to automated testing strategies; Record and Replay, Scripting, and Natural Language Programming (NLP), their pros and cons and ask ourselves “Is there a better way?”

Record and Replay

Let's start with Record and Replay. This approach allows the user to easily follow a natural journey through the application without ever having to think that they are actually "testing" it. The user doesn't need to know what commands to use, and it requires little or no training to start with automation. While recording, the user's full attention is focused on achieving the journey rather than automated testing needs.

Record and Replay is more precise with element selection. It can automatically capture a wide spectrum of deterministic selectors that will exactly match the element the user interacts with. But, it's also a more “organic” approach to test authoring because it's much closer to how a genuine user would likely behave. With Record and Replay, testers can simply interact with the application as if they were a real customer.


In testing, there will always be scenarios that are so exceptional and specific that codeless interfaces will not be able to address them. In these cases, you would be able to rely on the power of programming languages within an automation framework.

Being able to script means you are able to cover every single angle and detail inside your application scope as well as interact with external systems. Whether it's preparing the test environment conditions or validating that a third party received and processed an event triggered by the application, the number of practical applications for scripting is near incalculable.

The way scripting can be designed as a library for reusability and its content easily edited makes it even more flexible and versatile for your needs. The control and accuracy that a user has over interactions with the application, environment, and external systems is probably the strongest advantage of scripting.

However, the downside of scripting is simply the cost. Development cost is significant because the user needs to know how to code. He/she also needs to understand the basic principles of software engineering to be able to address different systems and environments.

The maintenance cost is also high because when the application changes, the scripts are often no longer valid, fail to execute, and require fixing. Because of these disadvantages, the use of scripting should be a last resort when everything else fails.

Natural Language Programming 

NLP testing is also known as "codeless test automation," but it's a phrase that many testers shy away from since codeless automation has been notoriously unreliable in the past. But hear us out before you pass judgment; codeless automation has come far in recent years, and now you can have the usability and accessibility of codeless along with all of the power of scripted automation. 

Since NLP tests are deliberately written or converted into plain English, they are more likely to be readable. Also, since users have already seen the tests once, they'll recognize them more easily and require less time to inspect them in the future. Studies in software testing support this (see a paper published by our own Head of Engineering, Sina Shamshiri) Using NLP also gives users the ability to write automated tests regardless of whether the current application version has been deployed or not. This gives users the power to effectively shift left and start building test automation earlier in the development phase because the tests do not depend on an exact version of the application.

Similar to scripting tools, NLP has increased reusability, control, and accuracy over the automatic recorders. These often have to guess user interactions and suffer when recording complex interactions due to corresponding technical challenges. With NLP, the user has full control and can describe exact user interactions.

Maintenance and augmentation of tests created using Record and Replay can be cumbersome, often requiring a replay of the journey. Meanwhile, in most cases with NLP, users can make the changes directly to their tests.

Let's not forget about portability, too. You can use a wide spectrum of technologies to write tests, from your smartphone or tablet to a simple piece of paper. For privacy-conscious users, with NLP there is no risk involved with installing an extension that can record everything they do and write.

Moreover, there is no dependency on a software application needing to be installed on your machine or browser for you to be able to write automated tests in plain English.


Record and Replay Testing vs Scripting vs Natural Language Programming

At Virtuoso, we ultimately believe the closer we can bring the authoring tools together, the better we can bring teams together and provide a richer user experience. Therefore, we support and offer all three approaches, seamlessly integrated with one another.

Here are some examples of how a hybrid approach could feature as part of a test automation strategy.

  • While recording using a Record and Replay (all done in-app, so no recording software required), the user sees their journey on the Live Authoring browser window on the application. They can manually write/change steps as they build the journey inside the application under test.
  • If a user partially writes a journey in NLP and runs it, the journey will continue recording from there using the Record and Replay.
  • While creating a functional journey that performs critical backend actions on one or more systems, the user may write a script library to make API calls to validate changes and data accuracy on backend and external systems.
  • Similarly, a user would be able to run a journey originally created in natural language on their extension, step through them with breakpoints, and make changes live when maintaining the tests. This can be really useful for when a journey breaks and the user has to first get to the target page before they can provide selector fixes etc.

In short, there is no one-size-fits-all approach to the testing process. The varied demands placed upon QA by dynamic applications and Continuous Integrations pipelines call for a mix of techniques, plus a testing tool that provides Continuous Testing. The reality is that some applications are more complex to test, as they integrate with more systems as part of complex business processes. Scripting will be required in part. But for the lion's share of testing, why script and maintain code when you can take advantage of speedier, more accessible ways of testing? It all comes down to need and preference.

What sets the codeless test automation tool that is Virtuoso apart from competitors is the fact that everything is combined into one platform. Whether you record it, script it, describe it in plain English, or let bots generate the tests for you in a sort of automated exploratory testing, Virtuoso takes your testing further with lightning-fast execution, maintenance, and self-healing capabilities.

Subscribe to our Newsletter