Restoration of lost information in digital signals

From historic speech and music recordings to the transmission of data, e.g. over a wireless connection, we frequently encounter the loss of significant segments of a signal. As with the settings for this kind of data loss, the reasons can be various, e.g. material defects or connectivity problems. As a consequence, we often have to work with data that is damaged and distorted, often to an unacceptable degree. By combining know-how from the areas of applied mathematics, optimization, signal modeling, signal and image processing and human auditory perception, MERLIN will provide new, innovative methods for the automatice recovery of lost signal segments and concealment of damaged signal content. In particular, MERLIN will devise methods specifically focused on audio restoration. In contrast to previous work in this field, the schemes designed in MERLIN will, whenever possible, harness existing signal-dependent information to assist the restoration process. This prior information can range from clues obtained directly from the signal, to meta-data on the signal source, e.g. instrumentation, speaker information and musical score. Audio restoration is a challenging and unique sub-field of signal restoration not only for the tremendous variety of audio signals and their everyday relevance, but also for the relevance of the inner workings of the human auditory system for their perception. To improve and simplify the restoration process, MERLIN will take into account state-of-the-art knowledge on the perception of sound. Together, these aspects will lead to significant advances in restoration quality. The algorithms designed throughout MERLIN will be supplied to the scientific community in a software toolbox, freely available for research purposes in conjunction with an extensive database of real and synthetic test signals.

Audio signals, such as speech and music recordings, are ubiquitous: We interact with audio signals in phone calls with our loved ones, online meetings with colleagues and customers, in entertainment media, but also, often unsolicited, in public transportation, stores and many other daily situations. The recording, storage and transmission of audio data is often prone to errors, which can, in the worst case, lead to the effective loss of significant segments of important data. The correction of such faulty segments is often referred to as audio inpainting, a terminology adapted from similar techniques in image processing, or more generally as error concealment. Both expressions correctly suggest that the exact recovery of the lost data is not always necessary, or even possible. Instead, the primary goal of audio inpainting is the production of a signal that is perceived as error-free. Apart from the mere correction of signal defects, audio inpainting offers notable opportunities for creative-artistic application. The project partners in MERLIN at the Acoustics Research Institute of the Austrian Academy of Sciences and the Signal Processing Laboratoy of Brno University of Technology were able to provide the theoretical and technical foundation, and implementations, of pioneering audio inpainting schemes. This success is based on fundamental research in the mathematical subjects of time-frequency analysis and harmonic analysis, and advances in the development of novel signal processing methods considering human auditory perception and exploiting cutting-edge machine learning and optimization schemes. Time-frequency representations are important tools for audio analysis and processing. Likewise, they are an essential component of successful audio inpainting methods, including those developed in MERLIN. This project provided the necessary foundation to draw on the full potential of adapted time-frequency representations, e.g., to human auditory perception, through the study of their fundamental, mathematical structure and properties. The use of such representations in matched, advanced optimization and machine learning schemes enabled the development of signal models for audio that are capable of describing the structure of such data more accurately than prior models which were often rather simplistic. The potential of these models was demonstrated by their use in MERLIN for the audio inpainting task, providing notable qualitative improvements over the state of the art.