Shift Right - The Value of Testing in Production
# How to monitor real time Customer eXperience (CX)
DevOps and CI/CD has changed the way we develop software and testing has had to embrace automation to keep pace with demand. And these new test automation platforms are offering more than just a stamp of approval.
We once understood testing simply as Quality Assurance, a manual, time-consuming, box-checking activity that is at odds with the automated, highly integrated approach to software development most teams now practice.
DevOps organisations have begun to develop a “Quality Assistance” attitude towards testing that keeps the development pipeline flowing through automation. What was once a bottleneck is now a facilitator of speed and quality. But the sophistication of these new tools continues to add value in post-production.
# The importance of testing live production systems
Today, customers and users of digital services and products now expect to receive a rich, faultless experience anywhere, on any platform, and at any time. Our tolerance for faulty software or a poor user experience has rapidly gone from low to zero.
And that’s not the only set of demands we face. The arrival of CI/CD and DevOps has contributed to a huge increase in software releases. There’s more to test and typically we’re never more than 2 weeks from making a release.
Taken together, this means QA has to facilitate pipelines that are releasing software faster and faster to end users who are increasingly impatient with downtime and poor performance. The value of ensuring consistently positive customer experience and site reliability has never been higher.
To actually understand production trends you need to put yourself in the customer’s shoes. What do they feel when accessing your application? How much time do they have to wait to complete a journey? What is their experience at a given point in time? Having this knowledge means having control.
# How to effectively test in production
Performing quality control by monitoring your production environment throughout the day allows you to measure application performance during peak hours. As well as noting any application downtime you can keep a close eye on functional health and application performance.
Being able to call upon quick responses to failures means you really have to understand how the system performs in response to real user behaviour. Seeing what your customers’ preferences and engagements are when applying a different application flow, or important design changes, will give you valuable knowledge
Traditionally, we build confidence by executing as many tests as possible prior to the release, but this approach has many flaws. It provides no assurance that the product will not fail in production because all the tests have been done within dedicated test environments.
These are some of the testing strategies that will help you to achieve effective CX monitoring.
- Core user journey testing: Execute a subset of journeys in the production system that cover the most important functionality.
- Run the tests frequently: Build a trend and see when and where faults or performance degradation occur
- Run your functional tests alongside performance tests: Execute load tests in pre-production to collect key indicators as to how load impacts user journeys. This should tell you what to monitor during peak user times in production
- Different geographies: Execute core tests from different locations to simulate connectivity issues and mimic real customer experience.
- Fault tolerance testing: Determine if the system under test is still able to operate under failure conditions on the critical functionality of a production system.
- A/B testing: Compare the feedback from real users on two versions of the application functionality and relates with the same comparison on the two versions’ performance.
- Canary release: Slowly roll out the change to a small subset of instances before applying it to the entire infrastructure.
# Reacting to issues in production
In DevOps organisations particular attention is paid to the ability to continuously shift right by testing in production. And when failures happen they act by deploying only small changes, released incrementally while the health of the application is closely monitored. It isn’t just about performance monitoring either. New, intelligent test automation platforms allow us to do much more in terms of functional testing and CX than is being done today.
Above all, it is important to be able to react quickly throughout the testing process and to maintain control of live situations. Testing in production makes it possible to understand real user journeys and analyse real use cases that are extremely hard to predict.
In the end, the only way to prove that software is ready for production is to release it into production! Automation in testing is giving us the freedom to do so with confidence.