Abstracts

This presentation focuses on the automation of regression testing based on AI -assisted usage analysis in operation. We will describe the concepts, the process from acquiring anonymous usage data, and the visualization of usage traces to the generation and execution of automated test scripts. The focus is on functional regression testing of web applications in a DevOps and continuous testing context. Two full-scale experiments of this “Usage -driven test automation process” are analyzed. Each focuses on a different objective:

1. SEPHORA: end-to-end regression tests on the eCommerce site
2. GENERALI France: regression tests of usage scenarios on the partner APIs of their platform. We performed these experiments in close interaction with the QA teams. The results will be made public at the JFTL conference in Paris on June 14 (presentations by François VLERICK - SEPHORA, and Mahdi GARSALLAH & Mathieu MEYERE - GENERALI France) - see cftl.fr website to access to the presentation (but in French). Our presentation is divided into four parts:

1- Motivation for usage-driven test automation - the problem addressed: the relevance of automated regression tests to real-life usage and the maintenance costs of automated functional tests are significant challenges for Agile teams. This complicates the transformation to DevOps and continuous testing.

2- The process from usage analysis to test automation:

• collecting anonymous usage data in compliance with the GPDR,
• measuring and visualizing actual test coverage by comparing usage traces and test execution traces,
• completing test coverage by selecting usage traces to be covered, and
• generating & running automated tests on GUI or API.

3- Machine Learning (ML) techniques we experimented with usage and testing data: clustering traces to analyze the traces, anomaly detection on usage traces with unsupervised ML, and prioritization of test execution by ML models from past execution data.

4- Feedback from the field - SEPHORA and GENERALI France: what was their motivation and the context of implementing the approach? What was the obstacle in the data acquisition process, and what was the gain for tests' relevance? What was the gain for test automation? We conclude with a summary of how regression test generation from usage is a step towards autonomous software testing capabilities (for regression testing).

Style of the presentation
The target audience is Testers and Test Managers. The presentation will be concrete and factual. We will not detail the Machine Learning models used but rather the practical issues of acquiring and using anonymous usage data for regression test automation. The focus is on the response to the new challenges of the Agile transformation, with the acceleration of the release pace and the need to implement a Continuous Integration - Continuous Testing - Continuous Deployment approach.