Model selection and inference with sparse models when the true model need not be sparse

Many of the most challenging problems in contemporary applied statistics feature a large number of potentially important factors or variables and a comparatively small number of observations. Examples include data from genomics, proteomics, mass spectrometry, or finance, to name a few. These problems have prompted the development of methods that find sparse submodels of high-dimensional overall models, i.e., models that include only a very small number of the many available potentially important factors or variables. The most prominent and successful of these methods is the LASSO and its variants. Also, methods for inference based on LASSO-estimators are currently being developed. Most theoretical results on the LASSO and its variants rely on some sort of sparsity assumption, to the effect that the majority of the available factors or variables are either irrelevant or that their influence is negligibly small. If such sparsity assumptions are violated, the performance of the LASSO is typically unknown. From an applied perspective this is problematic, because sparsity assumptions are usually impossible to verify in practice. The goal of this research project is to develop methods for model selection and subsequent inference with sparse working models without relying on sparsity assumptions.

The project has broken new ground in the area of predictive inference with shrinkage estimators. In particular, it was found that these methods can perform particularly well in situations where the system of interest is very complex and where, at the same time, the available training dataset is comparatively small. Such situations are very common in certain Big Data applications. For such situations, new prediction methods and new methods for predictive inference were developed.