Crossmodal metaphors in timbre cognition

Imagine yourself listening to a recording of the famous violinist Yehudi Menuhin (19161999) performing on his Khevenhüller Strad. How would you describe the sound of the violin, or the sound of Menuhin, or the sound quality of the recording? Timbral qualities of sound are often conceptualized and communicated through readily available sensory attributes from other modalitiesfor example, a sound seen as bright, felt as warm, or tasted as sweet because humans lack a sensory vocabulary for auditory experiences. This exemplifies a particular aspect of human cognition known as crossmodal correspondences: people make many synesthetic-like crossmodal associations between sensory experiences in different modalities (e.g., high frequency is mapped to high elevation, open vowels are matched to rounded shapes). Two questions are at the heart of theories concerning both the sensory- semantic description of timbre and crossmodal correspondences: 1) Which acoustic correlates of timbre drive crossmodal associations between sound and other sensory experiences and 2) are crossmodal mappings a result of direct communication between modality-specific sensorimotor systems or of supramodal (i.e., modality-independent) conceptual representations? This project will address these questions by: (i) developing a parametric synthesis model of sounds that lack readily available source-cause associations to ensure that the relevant intrinsic dimensions of timbre are precisely controlled and understood; (ii) using virtual reality forms (visual modality), palpated surfaces (tactile modality), and tasted edibles (gustatory modality) designed along scales with perceptually equidistant grades to facilitate the matching of auditory-nonauditory sensory experiences that evoke the same conceptualizations; and (iii) quantifying the conceptual relatedness between sound and other modalities in terms of systematically shared mappings between the position of a sound in timbral continua defined by the model and the position of a nonauditory stimuli on the corresponding scale. This will be the first systematic investigation of crossmodal correspondences between timbral qualities of sound and perceptual dimensions of other sensory modalities, bringing a new perspective into addressing old questions about the perceptual and cognitive processes underlying the way we talk about sound. Results will have an impact on diverse disciplines where timbre raises important issues, such as music, audio engineering and branding, acoustics, human and machine hearing, psychology, and aesthetics. At the same time, timbre will provide a testbench for understanding crossmodal correspondences and human semantic processing in general.