Submitted Abstract
Self-learning software systems (SLS) are integrated into a variety of domains ranging from safety-critical applications (autonomous cars and healthcare) to business-critical applications (finance, smart factories). Engineering such systems, however, is still a new practice, often not well-understood by engineers, and thus error-prone. It is therefore essential to provide engineers with means to assess that the SLS they build work reliably and as expected. In this project, we aim at complementing state-of-the-art machine-learning evaluation processes with testing techniques specifically adapted to the peculiarities of SLS. Indeed, although a plethora of techniques exists for testing traditional software, these are heavily challenged by SLS, their intrinsic probabilistic nature, their vast number of parameters, and their use cases too numerous to be elicited. More precisely, we focus on testing their underlying learning models and target three objectives: (1) measuring the adequacy of existing test cases with criteria that indicate how well the test cases cover the learning model; (2) defining model transformations (mutations) to modify the models, and estimating their sensitivity; (3) designing differential testing methods to discover disagreements between models, thereby obtaining new test cases that reveal errors in the models. Our three objectives are certainly not independent as fulfilling one will help achieve the others. Thus, altogether they will form a triangular chain of techniques to generate a high-quality test suite for learning models.