How to monitor real-time Customer eXperience (CX)

Continuous Integration/Continuous Deployment (CI/CD) has changed the way we develop software, and testing has had to embrace automation to keep pace with demand. And these new test automation platforms are offering more than just a stamp of approval.

We once understood testing simply as Quality Assurance: manual testing that's a time-consuming, box-checking activity that is at odds with the automated, highly integrated approach to software development most teams now practice.

DevOps organizations have begun to develop a “Quality Assistance” attitude towards testing that keeps the development pipeline flowing through automation. What was once a bottleneck is now a facilitator of speed and quality. But the sophistication of these new tools continues to add value in post-production.

The importance of testing live production systems

Today, customers and user bases of digital services and products now expect to receive a rich, faultless experience anywhere, on any platform, and at any time. Our tolerance for faulty software or a poor user experience has rapidly gone from low to zero.

And that's not the only set of demands we face. The arrival of CI/CD and DevOps teams has contributed to a huge increase in software releases. There's more to test, and typically we're never more than two weeks from making a release.

Taken together, this means QA has to facilitate pipelines that are releasing software faster and faster to end users who are increasingly impatient with downtime and poor performance. The value of ensuring consistently positive customer experience and site reliability has never been higher.

To actually understand production trends, you need to put yourself in the customer's shoes. What do they feel when accessing your application? How much time do they have to wait to complete a journey? What is their experience at a given point in time? Having this knowledge means having control.

How to effectively test in production

Performing quality control by monitoring your production environment throughout the day allows you to measure application performance during peak hours. As well as noting any application downtime, you can keep a close eye on functional health and application performance.

Being able to call upon quick responses to failures means you really have to understand how the system performs in response to real user behavior. Seeing what your customers' preferences and engagements are when applying a different application flow or important design changes will give you valuable knowledge. Traditionally, we build confidence by executing as many tests (all kinds of tests, from performance tests to integration tests) as possible prior to the release, but this approach has many flaws. It provides no assurance that the product will not fail in production because all the tests have been done within dedicated test environments.

These are some of the testing strategies and types of testing that will help you to achieve effective CX monitoring:

• Core user journey testing: Execute a subset of journeys in the production system that cover the most important functionality.
• Run the tests frequently: Build a trend and see when and where faults or performance degradation occur.
• Run your functional tests alongside performance tests: Execute load testing in pre-production to collect key indicators as to how load impacts user journeys. This should tell you what to monitor during peak user times in production.
• Different geographies: Execute core tests from different locations to simulate connectivity issues and mimic real customer experience.
• Fault tolerance testing: Determine if the system under test is still able to operate under failure conditions on the critical functionality of a production system.
• A/B testing: Compare the feedback from real users on two versions of the application functionality and relates with the same comparison on the two versions' performance.
• Canary release: Slowly roll out the change with canary testing to a small subset of instances before applying it to the entire infrastructure.

Reacting to issues in production

In DevOps organizations, particular attention is paid to the ability to continuously shift right by testing in production. And when failures happen, the product team acts by deploying only small changes, released incrementally while the health of the application is closely monitored on a daily basis. It isn't just about performance monitoring either. New, intelligent test automation platforms allow us to do much more in terms of functional testing and CX than is being done today.

Above all, it is important to be able to react quickly throughout the testing process and to maintain control of live situations. Testing in production makes it possible to understand real user journeys and analyze real use cases that are extremely hard to predict.

In the end, the only way to prove that software is ready for production is to release it into production! Automation in testing is giving us the freedom to do so with confidence.