Synaesthesia - A Cognitive Model of Cross Modal Association

Andrew D. Lyons.

Composition Unit

The Sydney Conservatorium of Music.

The University of Sydney

Sydney NSW 2000 Australia

http://www.vislab.usyd.edu.au/user/alyons

Email: alyons@vislab.usyd.edu

Abstract

The cognitive characteristics of the rare perceptual condition known as synesthesia provides a clinical insight into the relationship between the various human sensory modalities and in particular for the relationship between audition and vision. Following a discussion of the nature of synesthetic perception, this nature is discussed within the context of the relationship between auditory and visual arts.

1 Introduction

Traditionally, the arts have been separated into disciplines delimited by medium and other criterion. Painting and music for example are delimited by, amongst other things, the different senses by which they are perceived - we hear music and see painting. One of the great dreams of the romantic tradition has been that works particular to each artistic discipline might be meaningfully represented in another artistic discipline. One of the great challenges to the inter-disciplinary translation of artworks has been the development of a system of mapping perceptual attributes between each of the five human senses. Whilst mapping between sculpture and painting may be achieved in a very literal way, mapping between music and painting has always presented itself as more of a challenge. A study of human perception, especially as it pertains to the relationship between audition and vision, can prove very useful toward this end.

2 Clinical Synaesthesia

2.1 Human Sensory Systems

"Human sensory systems mediate four attributes of a stimulus that can be correlated quantitatively with a sensation: Modality, intensity, duration and location." [1] The attributes of intensity, duration and location apply to all five sensory modalities: vision, hearing, touch, taste and smell. Each of these sensory modalities has sub modalities, which in the case of vision include color whilst in hearing they include pitch. Our perception of light arrives to the brain via a series of Photo-receptive rods and cones in the eye. Audition on the other hand uses information gathered by mechano-receptive hair cells in the ear that measure vibrations in air pressure. The nature of the differences between the five modalities is suggested by the disparate nature of these sensory receptors.

Whilst both photo-receptors and mechano-receptors measure intensity, location and duration, they both also measure a property of frequency. Pythagorus, Sir Isaac Newton, and numerous other physicists, have hypothesised about the existence of a physical relationship between the frequencies of light and sound responsible for the sub-modalities of colour and pitch. However, explanations of the relationships that exist between sensory sub-modalities, have been made in more recent times by Psychologists and Neurophisicists.

2.2 Definition of Synesthesia

"Synesthesia is a word derived from the Greek words 'syn' meaning together and 'aisthesis' meaning perception. It is used to describe the involuntary physical experience of a cross-modal association. That is, the stimulation of one sensory modality reliably causes a perception in one or more different senses." [2] For example, a synesthete will see coloured shapes projected into their field of vision as a result of auditory stimulation.

2.3 Does Synesthesia Exist?

Synesthesia has in the past been considered a less than scientific area of research by some, due to its reliance on subjective sources for any observations of its nature. Scientists have recently been convinced of the existence of synesthesia and cite evidence in support:

• The impressive test-retest reliability in the consistency of colours triggered by different words (in the case of "coloured hearing").
• The similarity of reports from different cultures and different times across the century.
• The consistency of sex ratio (it is overwhelmingly a female condition).
• The familial pattern to the condition.
• The neuroimaging data (using PET) showing different cortical blood flow patterns in women with synaesthesia in comparison to women without the condition.

2.4 Clinical Diagnosis of Synesthesia.

Psychologists and more recently Neuropsychologists have documented the nature of synesthetic experience in a useful manner for over a hundred years. Varying criteria has been applied to the diagnosis of synesthesia although in general psychologists have always differentiated clinical synesthesia from metaphor, literary tropes, sound symbolism, and deliberate artistic contrivances that sometimes employ the term "synesthesia" to describe their multi-sensory joinings. Dr. Richard Cytowic has proposed five criteria for the diagnosis of a type of clinical synesthesia called ideopathic or developmental synesthesia as opposed to acquired forms of clinical synesthesia such as drug induced synesthesia, epileptic synesthesia, and synesthesia due to acquired brain lesions:

• Synesthesia is involuntary but elicited.
• Synesthesia is projected. If visual, a photism will appear outside the body in the region close to the face.
• Synesthetic percepts are durable and discrete. The associations for an individual Synesthete are stable over their lifetime. If a sound is blue, it will always be blue.
• Synesthetic experience is memorable. Many synesthetes exhibit hypermnesis.
• Synesthesia is emotional in nature. A synesthetic experience is accompanied by a sense of noetic certitude.

2.5 Non Uniformity in Synesthetic Perception.

In addition to these characteristics it should be added that there is no uniformity amongst the experience of synesthetes. Each individual experiencing synesthesia experiences it in a unique form. "In fact, this rather glaring problem - that two individuals with the same sensory pairings do not report identical, or even similar, synesthetic responses - has sometimes been taken as 'proof' that synesthesia is not 'real.'" [3] Yet it remains that certain patterns have remained constant in the statistical information derived from scientific observation of synesthetic perception. Some of these patterns, such as the correpondance between pitch and visual brightness, have been documented repeatedly since they were first described in the experiments of Bleuler and Lehmann in 1881.

Lawrence E. Marks describes the situation thus: "One should not come away with the impression that all our knowledge about our sensory and perceptual experiences can be captured in a set of independent - or even interrelated - verbal categories; nor that sensory/perceptual experiences themselves reduce in any simple manner to a list of attributes... Still, the study of synesthetic metaphor may serve as a useful model system. By being amenable to psychophysical analysis, synesthetic metaphors not only permit ready quantification, but enable us to assess development trends in the ways that at least certain aspects of such metaphors are interpreted... A psychophysics of synesthetic metaphor as described here may eventually reveal much about perception and language; but to appreciate the depth and extent of human metaphorical capacity will demand a psychological analysis that is as yet hardly dreamt in our philosophy."

2.6 Explanations of Synesthetic Perception.

Over the past 200 years a number of hypotheses have been put forward to explain the cause of synesthesia. Current theories however in some way recognise the findings of recent neurological studies that suggest the possibility that the executive areas of the human brain, primarily in the frontal lobes, manifest a high degree of sensory integration. The Cross-Modal Transfer (CMT) hypothesis is now a widely accepted explanation for the occurrence of synesthesia although it was radical when it was first proposed. The CMT hypothesis supports the view that detection of intersensory equivalence is present from birth, and that perceptual development is characterized by gradual differentiation.

2.7 The Neonatal Synaesthesia hypothesis

The Neonatal Synaesthesia hypothesis builds on the CMT evidence, but suggests that early in infancy, probably up to about 4 months of age, all human babies experience sensory input in an undifferentiated way. Sounds trigger auditory, visual and tactile experiences all at once. Following this early initial phase of normal synaesthesia, the different sensory modalities become increasingly modular. Adult synaesthesia, has been suggested to be as a result of a breakdown in the process of modularization, such that during infancy the modularization process was not completed. This of course implies that if not now, then at some time in the past, we have all experienced synesthetic perception.

3 Synesthesia and Art

3.1 Photisms in Coloured-Hearing Synesthesia.

In coloured hearing synesthesia, a photism, usually coloured in some way, appears in the field of vision of a synesthete as a response to some form of aural stimuli. Synesthetic photisms usually vary in shape and color according to the nature of the stimuli that triggered them. The examples below represent some different photisms.

As can be seen above, "Synesthetes never see complex dream-like scenes or have otherwise elaborate percepts. They perceive blobs, lines, spirals, lattices, and other geometric shapes." [4] Dr Richard Cytowic notes that the generic and restricted nature of synesthetic percepts bear a considerable likeness to a series of forms first developed by Heinrich Kluver in the 1920's known as Kluvers "form constants". [5] These generic shapes are common to synesthesia, hallucinations and are frequently seen in primitive art. "

Figure 2.2 Kluver's Form Constants

Variations in photism color, brightness, symmetry, and shapes have been recorded to vary as a result of variation in musical stimuli. Tempo for instance effects the shape of a photism; the faster the music, the sharper and more angular the photism. That pitch has a direct effect on the size of a photism has also been recorded. It has been observed universally that photism size increases as auditory pitch decreases. In this way high pitched sounds produce small photisms and low pitched sounds produce synesthetic percepts that are large in size. Loudness also has an effect on the size of the photism perceived by a synesthete.

Lawrence E. Marks shares his understanding of Synesthetic response to music :"Just as the important dimensions of the auditory stimulus that are responsible for musical synesthesiae can be quite complex, so too can be the synesthetic responses themselves." "Visual sensations aroused by music need not be limited or confined to simple spots of color. Often the entire visual field fills with colors that change over time with the music; some subjects report several colors simultaneously, each color reflecting a particular aspect of the music." [6]

3.2 Musical Perception in Chromaesthesia.

It is of interest to open this section by quoting the concerns of one group of psychologists who conducted numerous investigations into synesthesia during the first half of this century. Published in The Journal Of General Psychology in 1942, they write, " Although it is generally agreed that relationships between visual and auditory experiences exist commonly in our language forms, nevertheless, we have no quantitative measure of just how common a given relationship between sound and sight actually is in the population. Such a measure would be useful to determine what per cent of an audience could be expected to grasp an artists purpose if , for example, he represented the harmony of his musical composition by background and the melody by figures in his color-music production. Also it would facilitate the process of conventionalising associations between music and vision if one could determine quantitatively which of several acceptable ways of representing a melody, for instance, is already predominately in use-i.e., is considered appropriate by most people." [7]

At least 23 psychological publications between 1862 and 1974 concern themselves with correlations between sound composition and colour as a result of research directed at synesthesia. Many more have concerned themselves with studies of synesthesia triggered by speech stimulus. Research into these areas of synesthesia has furnished artists with some information by which to start developing the formulations suggested in the quote from 1942 above. In his article, On Coloured-Hearing Synesthesia originally published in the Psychological bulletin, Lawrence E. Marks compiled all the information extant on such matters into a series of tables printed below: [8]

3.3 Vowel Colour

The study of chromesthetic phenomena often concerns itself with associations triggered by speech rather than music. This is perhaps due to the fact that speech is pathologically superior in its ability to evoke a synesthetic response. The component of speech that bears the greatest influence on the nature of the induced response is the sound of vowels. Both areas have tremendous significance in mapping out perceptual parallels between the modalities of hearing and vision. Firstly, when it comes to reports on musical synesthesia, we find that the important principles of visual-auditory association that manifest themselves in color music are basically the same principles that manifest themselves in coloured vowels - that is, the relations of visual brightness and size to auditory pitch and loudness. Secondly, in an article published in 1968, Wayne Slawson showed that artificial two formant sounds are readily interpretable as vowels and as musical notes and that the vowel quality and musical timbre depend in similar ways on the structure of the sound (formant frequency and spectrum envelope.)

3.4 Slawsons' Sound Colour

Slawson went on to elaborate his comparison of vowel sounds to the field of musical timbre in his Sound Color of 1985. In this book, Slawson uses the four characteristics set out by Chomsky and Halle in 1968 in The Sound Patterns of English and adapts them so as to form a basis for organising musical timbre derived from varied sources. Besides being the only text to date to take up Arnold Schoenbergs' 1911 request for a treatise on the subject of KlangenFarbe, Sound Color forms an essential bridge between the colors commonly associated with vowel sounds and the formant composition of synthesised sounds with which Slawson is largely concerned. Slawson indicates that sound colour is primarily a function of the frequencies of the first two resonances. Slawson uses the three categories by which vowel features are organised - compactness, acuteness and laxness, changing compactness to openness, and adding a fourth category - smallness; which has no corresponding vowel feature.

4 Conclusion

The nature of Synesthetic perception does not on its own provide artists with a template for mapping between visual and auditory arts. Whilst it describes the constant nature of the relationship between brightness and pitch, brightness and volume, photism shape and aural texture - it does not map hue to timbre or pitch in any way not related to a particular musical context. Also, because synesthetic photisms are two dimensional in nature, research into synesthesia can shed limited light on relationships between music, and objects with three dimensions.

5 Footnotes

[1] John H. Martin, "Coding and Processing of Sensory Information" in Principles of Neural Science, ed. Eric R Kandel, James H. Schwartz and Thomas M. Jessel. (London: Prentice Hall, 1991), 329.

[2] Richard E. Cytowic, "Synesthesia, phenomenology and neuropsychology: a review of current knowledge," Psyche 2.10 (1995), 1.

[3] ibid.

[4] Richard E. Cytowic, Synesthesia: a union of the senses, (New York: Springer Verlag, 1989), 138.

[5] Kluver, H. Mescal and Mechanisms of Hallucinations, Chicago: University of Chicago Press, 1966.

[6] Lawrence E. Marks, "On Coloured-Hearing Synesthesia", " in Simon Baron-Cohen and John Harrison, eds. Synesthesia: Classic and Contemporary Readings, (Oxford: Blackwells, 1996), 70.

[7] Karwoski, T. F., H. S. Odbert and Charles E. Osgood. "Studies in Synesthetic Thinking II: The Role of Forms in Visual Responses to Music." Journal of General Psychology 26 (1942) : 205.

[8] Lawrence E. Marks, "On Coloured-Hearing Synesthesia", " in Simon Baron-Cohen and John Harrison, eds. Synesthesia: Classic and Contemporary Readings, (Oxford: Blackwells, 1996), 70.

6 Bibliography