Audio-tactile short-term memory

We humans perceive holistically. If we set the string of a cello vibrating, for example, the audible and the tangible (here, the vibration of the string, generally the tactile) impressions blend. This improves our perception: Congenital or object-related inaccuracies or weaknesses in one sense, for example, the auditory sense, can be compensated for by more accurate or stronger processing in another, for instance, the tactile sense. If one hears a vibration badly or not at all, for example, because it is noisy, one can still feel the type of vibration of the cello string - its frequency and its strength. The overall goal of the project is to examine the processing steps of the human mind and brain that enable the blending or integration of heard and felt impressions. In general, this integration occurs in a form of short-lived memory: short-term memory, which encompasses the most recent sensory impressions. Our research is expected to provide insights that can inform the development of even more effective sensory prostheses, such as hearing aids that use tactile sensing in addition to hearing. Our theoretical starting point is a new model of audio-tactile short-term memory. It has a kind of internal clock that marks simultaneously arriving features that are pre-processed in different brain areas as belonging to the same point in time. By virtue of this marking, our brain knows: These features from different senses processed in the brain occurred at the same time, belong together, and must be integrated into one impression. We use laboratory experiments examining human performance in hearing and feeling to test the model. Based on the inner clock model, precise predictions are made, for instance, that people should rely more on their tactile sense if they currently hear less. By repeatedly comparing experimental and predicted data, errors in the model will be identified and eliminated step by step. The project comprises basic research on audio-tactile short-term memory and tests new theoretical assumptions, for example, which sensory impressions are still perceived as simultaneous despite being asynchronous. Furthermore, it provides the basis for applications, such as the development of intelligent algorithms for audio-tactile hearing aids. Our project is financed by the Austrian Science Fund FWF (budget: EUR 235,623.79) and is an interdisciplinary cooperation of the University of Vienna (Institute for Psychology of Cognition, Emotion, and Methods; Ulrich Ansorge, Paul Seitlinger, Marie-Luise Augsten) and the Austrian Academy of Sciences (Acoustics Research Institute; Bernhard Laback).

Title. Tactile and Auditory Short-Term Memory and Perception Wider research context / theoretical framework. The project complemented existing models of human tactile Short-Term Memory (STM) to also explain effects of tactile and auditory perception. To this end, we draw on the Retrieved Context Theory for a novel cross-modal STM model and assume information of successive items (e.g., tactile vs. auditory perceived frequencies) to blend into one evolving context signal: The bindings formed between items and their contexts are used for later retrieval of item information, e.g., to compare frequencies from memory. Core characteristics of the novel STM model are (1) an integration of different sources of memory interference; and (2) a mechanistic explanation of statistically optimal decisions based on different modality-specific item layers and one shared context layer. Objectives / hypotheses. Part I of the present project focused on tactile STM. The main hypothesis was that interference in tactile frequency memory arises from an increased blending of context states and, thus, context-item binding errors can occur. In Part II, we wanted to extend the model to a separate modality, audition. We have tested a model version including context-item bindings for features and temporal positions. Approach / methods. The hypotheses of the tactile modality were examined in a series of four experiments (two in each Part), applying variants of a Delayed Match-to-Sample (DMS) task and a reproduction method. Model-based simulations and analyses of DMS data were used for detailing and stringently testing theoretical assumptions about the processing of tactile STM representations. In a further experiment, data were also collected in the auditory domain (in Part II). Level of originality / innovation. A novel, more realistic STM model was developed to better explain (1) tactile memory performance and (2) auditory performance. The model and its tests address important open research questions about tactile STM, a still under-researched domain in cognitive psychology. The extended and tested version of the model sheds fresh light on human tactile perception. The project is timely under an applied perspective: A valid model of human tactile perception should be useful for innovation in applications in this domain, like forced tactile feedback. Primary researchers involved. Prof. Dr. Ulrich Ansorge, Dr. Paul Seitlinger, Doz. Dr. Bernhard Laback