Architecture Support for the Testing of Complex Software Systems
by Antonia Bertolino and Paola Inverardi
We describe a new research project aimed at investigating methods and criteria for architecture-based software testing. The main partners in this collaboration are IEI-CNR, with several years of research experience in software testing, and the University of L'Aquila, well established in the field of software architectures.
The engineering of modern software systems is very hard. A principled use of Software Architecture (SA) in the design stage, when most crucial choices are made, helps to overcome system complexity, and hence can be a determining factor in development success. At a high level of abstraction, a Software Architecture provides an explicit and formal description of both the static and the dynamic properties of a system, ie the components and connectors. Our interest in the field of Software Architecture is in promoting the use of rigorous, formal descriptions of Software Architectures during development and, in particular, in order to improve analysis and testing.
The nature of research in Software Architectures makes it necessary, if not essential, to experiment on real-world systems. The scale of the applications considered is in fact one of the key aspects of the problem: the technical issues that arise when testing at the architectural level cannot be studied using simplistic, toy programs. Our project will thus involve several companies interested in improving the cost-effectiveness of their testing processes. These companies will act mainly as case study providers, but will also help us to keep the focus on substantial (and not purely academic) questions and will check the practicality of proposed solutions.
One case study is being provided by Hewlett Packard Italiana S.p.A., and consists of a network scanner. We are investigating methods for deriving an integration test plan, based on a formal description of the SA of the scanner. The main challenges of this case study are to guarantee portability over a multitude of platforms, and to meet the strict deadlines imposed by market competition. A second case study is provided by the Parco Scientifico e Tecnologico d'Abruzzo, a regional consortium of public and private research institutions and manufacturing industries. The system under development here is a Teleservice and Remote Medical Care System, which poses several safety and reliability constraints. We plan to involve other industries in the future.
Testing is one of the most expensive activities in software development and is mandatory to improve and evaluate the dependability of complex software systems. Traditionally, the testing of such systems proceeds in a series of successive phases, known as unit, integration and system tests. This step-wise approach on one hand allows testers to master the inherent complexity of the task, by subdividing the test object into smaller, more manageable pieces, and, on the other hand, enforces a disciplined procedure. However, we have identified several areas for improvement:
- It is often stated that the goal of the integration test is to check the compatibility between component interfaces. But static compatibility between interfaces is only one dimension of the problem: the design of software components naturally embodies assumptions about both the structure and the behaviour of the contexts in which they are to be integrated. Conflicts between components can thus be of a static nature, as well as dynamic. Conventional integration testing stresses structure over behaviour and is not well suited to uncovering dynamic conflicts.
- Automation is difficult and expensive, because the selection of test cases for the subsystems, and the sequencing order of the selected tests, depends on the functionality decomposition of the adopted system, which is not formalised. A new integration test strategy has thus to be re-invented ad hoc for each new system.
- When existing components are re-used in the integration of the system, as is often the case for large, complex systems, the test results obtained from the integration testing of the same components within different systems are not re-usable. The reason is that the role of components within the system, and the extent to which they are being tested, are not formally expressed. Given the high cost of test selection and execution, a capability to recover earlier test efforts would be highly desirable.
Our research activity is directed to addressing these problems by using the SA of the system as a primary reference. SA has recently emerged as an explicit field of study for software engineering researchers and practitioners. SA provides a disciplined basis for the engineering of large systems constructed out of many interacting components. In particular, since an SA provides an explicit and formal description of both the static and the dynamic properties of a system, it provides an important, if not essential, support towards making integration testing a more systematic and automatizable process. Not only do we expect to find suitable and workable solutions to the problems described above, but we also envisage several side benefits. For instance, when the design of an SA is based on the adoption of a specific architectural style, which constrains the components and the connectors to a restricted list, the testing itself can be tailored to the adopted style. We thus intend to investigate interesting architecture test patterns (in strict relationship with existing design patterns). Moreover, an SA typically specifies information about the extra-functional properties of components and connectors as well. Therefore, we also intend to investigate how to exploit the architecture specification to test for extra-functional requirements.
Please contact:
Antonia Bertolino - IEI-CNR
Tel: +39 50 593478
E-mail: bertolino@iei.pi.cnr.itPaola Inverardi - Universita' di L'Aquila
Tel: +39 862 433127
E-mail: inverard@univaq.it