Regularization Techniques in Learning with Big Data

Methods of the Regularization Theory had been introduced in the context of learning in the beginning of 90s by the seminal works of Poggio and Girosi. Due to the nature of the learning tasks, the regularization techniques, such as Lavrentiev regularization, which were originally developed for dealing with continuous ill-posed problems, have to be applied to discrete data. But because of the intrinsic ill-posedness of the learning problems and the risk of the so-called overfitting, a discrete form of the data does not make the application of the regularization methods easier for the analysis and interpretation. Therefore, there is a substantial literature on the regularization based learning, and research in this area is still very active. At the same time, Big Data Learning becomes a hot research area because of its great potential and importance in many scientific and industrial contexts. However, a straightforward application of the known regularized learning algorithms may become intractable for learning from big data. In this project, we argue that the intractability issue can be addressed by using new ideas emerged in the regularization theory.

An important class of machine learning algorithms is formed by the so-called kernel methods operating in the Reproducing Kernel Hilbert Spaces, where the process of determining output values of a function is well-defined for any input. However, the standard implementation of these algorithms is infeasible when dealing with big training data because it exceeds the computational capacity of the conventional computers. One of the achievements of the accomplished project is the development of a fully data-driven strategy for the simultaneous tuning of the hyperparameters of the kernel learning algorithms that allows their feasible implementation with no loss of accuracy order compared to the ideal use of the available big training data sets. To facilitate the testing of the theoretical results against real world data sets, the project was transferred to the Medical University of Innsbruck, where the use of the machine learning algorithms in the neuroradiological data processing was also promoted. The main results of the project have been published in the leading scientific journals in the fields of Computational and Applied Mathematics and Radiology. The project was performed in close cooperation with other project teams supported by different granting agencies. Such a synergy of research activities can be seen as an example of attempts to increase the value of the public money invested in research.