MATE-Model-Based Acceptance Testing

Software testing is one of the core fields in software engineering and is the essential technique for quality assurance in industrial software development. Surveys like a NIST study in 2002 reveal that the potential for improvements in the testing process is still enormous and can reach up to 30% of the costs of the testing process. In the recent years, the area of unit and component testing has been target of crucial innovations. Inspired by the ideas of agile software development the ideas of regression testing and strict separation of test code and system code have found their way into practice. Moreover, the development of model-based testing approaches has been an important step towards a gain of abstraction and tight coupling of test cases and requirements. In contrast, acceptance tests for a long time have been primarily performed manually, e.g. by the owner or user of the system under test. Acceptance or system testing is commonly understood as black-box testing performed on a system prior to its delivery. In many cases output of acceptance tests are test reports about the behaviour of the system which are produced manually by the tester. However, in the era of incremental software development non-automated tests are expensive and error-prone. Therefore a number of tools provide frameworks for executable acceptance tests. A critical issue for executable acceptance tests is to close the gap between the specification of test cases at a business-oriented level and the link to the code of the system under test. Acceptance test frameworks should provide concepts that enable clients to specify or at least to understand test descriptions. While there are quite many success stories in practical applications acceptance test frameworks so far do not exploit the huge potential that is provided by the information in the requirements specification. The major goal of the MATE project is to develop a tool-supported method for acceptance tests which on the one side are executable and support regression testing. On the other side acceptance tests shall be tightly coupled with the requirements specification and thus provide potential for automated generation of test cases and tool-supported check of consistency and coverage conditions. The MATE framework will be model-based which in our context means that the system under test is attached with a model of the requirements (called System Model) and the Test Model describes test cases in the language of the model elements of the System Model. Test cases are conceived at business level as exemplified interactions (scenarios) of the actors with the system. The major innovation of MATE is that it provides tight coupling between the requirements specification and the description of test cases and thus paves the way for specifying, checking and generating acceptance tests in a rigorous way.

The goal of Mate has been to provide a tool-supported methodology for model-based system testing of service-centric systems. Mate provides tight coupling of requirements and design artefacts with description of test cases and thus paves the way for specifying, checking, generating and executing system as well as acceptance tests of service-centric systems in a rigorous way. The fields of MBT (Model Based Testing) and MBSE (Model Based Software Engineering) have been significantly enhanced by providing a model-based testing approach which specifies tests on the model-level before or interwoven with system modelling: Mate extends model-based testing approaches like the UML Testing Profile (UTP) by providing tabular specification of test cases, and test-driven approaches like the Framework for Integrated Test (FIT) by providing early, abstract and processible model-based test specification. Furthermore, Mate supports security requirements testing as well as evolution of service-centric systems and tests on the model level. Mates model-based testing approach for service-centric systems realized the following key innovations:Mates Model-Based Testing Methodology. In Mate tests are represented as separate test models. The test models are designed in such way that they can be validated against the requirements model and the system model. Differing from other approaches, Mate can be applied in a test-driven way, i.e., the test models are defined prior to the behavioural artefacts of the system, and the test results are visualized in the test model due to traceability between all artefacts. The methodology is based on a novel testing process, which is test-driven, validates the models, directly transforms test models to executable test code, and integrates the test result into the test models.Mates Test Tool. The Mate methodology is tool-based. The tool implementation provides domain specific language for requirements, system and test modeling. The tool comes with validation checks based on Object Constraint Language (OCL) to guarantee the quality of the models. A test code generator transforms the test models to executable test code which is then executed by a test engine. The test results are visualized in test reports and annotated to model elements.Model-Based Security Testing with Mate. Mate has been applied to test functional and non-functional security requirements. Differing from other model-based security testing approaches a broad class of security requirements are testable. For testing non-functional security requirements risk-based testing approaches are considered on the model level.Model-Based Regression Testing with Mate. Mate supports state-based management of system and test changes, its integration into a testing process as well as the generation of regression test suites based on the system and test states. The regression testing strategy is configured by OCL statements.The Mate methodology and its tool implementation have been evaluated in three industrial case studies from the telecommunication, domotics and healthcare domain.