Das a.r.t.e.s. forum ist die interdisziplinäre Jahrestagung der a.r.t.e.s. Graduate School for the Humanities Cologne, die Wissenschaftlerinnen und Wissenschaftler aus dem In- und Ausland einlädt, aus ihrer jeweiligen disziplinären Perspektive zum Tagungsthema Bezug zu nehmen. Das a.r.t.e.s. forum 2016 versammelte Beiträge zum Thema "text – language – media". Im Folgenden ist der Bericht von Prof. Dr. Lars Elleström zu seinem Vortrag beim a.r.t.e.s. forum am 15. Juli 2016 zu finden.
Visual Iconicity in the Light of Media Modalities
Lars Elleström (Komparatistik, Linnaeus University) zum a.r.t.e.s. forum 2016
The starting point of my presentation was what I have called elsewhere the four modalities of media: the material, the sensorial, the spatiotemporal, and the semiotic modalities. A modality should be understood as a category of related media characteristics that are basic in the sense that all media can be described in terms of all four modalities. All individual media products, and most conceptions of media types, may be understood as specific combinations or varieties of combinations of modes of the four modalities, in other words, unions of particular traits belonging to the four modalities. The modes of the modalities far from cover all media characteristics but form a type of skeleton on which all media are built.
Thus, the four modalities of media and, more specifically, the modes of the four modalities, constitute a transmedial foundation that cannot be overlooked if one wants to understand the ways in which more complex media characteristics are represented. The flat surface, as a mode of the material modality, is an aspect of computer games, acrylic paintings, posters, printed literature, and so forth. The audible, a mode of the sensorial modality, is an aspect of instrumental music, recited poetry, motion pictures, weather forecasts on the radio, theatre, and many other qualified media types.
Temporality, a mode of the spatiotemporal modality, is an aspect of songs, speeches, gestures, and dance, but not of stills and sculptures. All media are perceived and decoded in time, which is not the same as being temporal in itself. However, some media types, such as most printed visual literature, are conventionally decoded in a fixed sequence, which makes them second-order temporal, so to speak.
Iconicity, a mode of the semiotic modality, is a vital aspect of the production of cognitive import in media such as children’s books, statistical graphs, music, and web sites. Iconic structures create meaning through resemblance. Although similarities are most clearly perceived among visual and auditory phenomena, respectively (a photograph of a boat clearly looks like a boat and a skilled whistler is able to sound just like a blackbird), similarities can be established across both sensorial and spatiotemporal borders. For example, visual traits may depict auditory or cognitive phenomena, and static structures may depict temporal phenomena; a graph may depict both changing pitch and altering financial status. In general, iconicity interacts with the two other main modes of semiotic modality: indexicality (based on contiguity) and symbolicity (based on habits).
Hence, the three modes of semiotic modality are identical to Charles Sanders Peirce’s three basic types of representation. Iconic, indexical, and symbolic signs are not in any way mutually exclusive; as modes of the other modalities, they are often combined to create multimodal media: media that are visual and auditory, spatial and temporal, iconic and indexical, and so forth.
The observation of differing modes of the modalities is necessary for pinpointing media similarities and differences, and for analyzing media combinations and media transformations. Whereas the material, sensorial, and spatiotemporal modalities form the framework for explaining presemiotic processes of mediation, the semiotic modality is the frame for understanding representation – the mental creation of meaning. In order to illustrate the advantage of observing the differences and interactions between media modalities – again understood as categories of related media characteristics – I then continued with focusing on the example of visual iconicity in poetry.
The fact that the term “visual iconicity” is anything but tautological cannot be over-emphasized, although visuality and iconicity are often connected. A refined understanding of the production of meaning in media requires a distinction between the sensorial and the semiotic. Iconicity, understood as meaning generated by resemblance, is an important semiotic mode and is certainly not restricted to the visual.
The issue of iconicity and language has been the subject of debate since classical antiquity. Speculative ideas about iconic traits of language led to skepticism, culminating with Ferdinand de Saussure and his followers, who claimed that iconicity is, at most, a marginal phenomenon in language. Although no one would dispute the fact that the basic language signs are primarily symbolic or conventional, systematic and empirical linguistic research during the 20th century has made it clear that iconicity is a multifaceted and widespread feature of language that reaches far beyond simple sound imitations. In isolation, visual and auditory traits of letters, syllables, and words are usually only marginally iconic; as parts of larger linguistic structures, however, their iconic potential increases.
Many iconic traits are combined with or even dependent on conventions, but that does not indicate the absence of iconicity. Also, what is taken to be conventional meaning is often based on iconicity. A drawing of a stick figure uses conventions—as such, the thin lines are rather dissimilar to head, trunk, arms, and legs—but the figure represents a human being also because of the resemblance between the overall forms of the figure and the body.
Contrary to Saussure, Charles Sanders Peirce made seminal explorations into the realm of iconicity around the turn of the 20th century. Peirce distinguishes between three constituents of the sign: the representamen (or simply the sign), the object (what the representamen refers to), and the interpretant (the final outcome “in the mind”); the representamen and the object are always connected in or even conceived by the mind in which meaning is generated. The representamen “stands for” its object but representamen–object relations are not complete or unambiguous; according to Peirce, whom I follow, signs are results of interpretive activity by an agent.
The complete sign function is thus the result of a dynamic mental activity. All signs are signs because we take them to be signs, and all signs are part of infinite chains of semiosis. The interpretant is not a simple effect of a representamen and an object that are plainly presented from the outside; the representamen and the object are actually results of an interpretive activity. However, it must be stressed that although representamens and objects are essentially mental entities, they may have strong material facets: signification may be triggered by material objects such as media products and it may connect to experiences of the surrounding world. This is actually the most common view on signs: they are physical entities connecting either to ideas or to other physical entities.
Consequently, iconic signs (based on similarity between representamen and object), as indexical signs (based on contiguity), and symbolic signs (based on habits or conventions), may have representamens that are either material or mental and that emanate from sensory input and mental activity, respectively. In any case, an iconic sign is a sign “by virtue of characters of its own”; it is a sign “in so far as it is like” the object it stands for. However, in order to understand iconicity in a more detailed manner, it is necessary to bear in mind a few more distinctions. Peirce also suggests that iconic representamens could be “roughly divided” into images, diagrams, and metaphors.
Unfortunately, his way of circumscribing three basic iconicity types is notoriously cryptic. I choose to understand these three notions as a spectrum of multifaceted similarity relations between representamens and object. The similarity relations should be judged on the basis of two inverted parameters: the relative strength of sensory resemblance on the one hand, and the relative complexity of cognitive operations involved in establishing iconic meaning on the other.
Accordingly, an image might be understood as an icon based on strong sensory resemblance, requiring only simple cognitive operations to be established. For instance, it is easy to see a tailor’s dummy as an image of how a human being looks. Inversely, a metaphor might be understood as an icon based on weak, cross-modal, or almost non-existent sensory resemblance, requiring complex cognitive operations. For instance, it takes rather intricate brainwork (even if it may come effortlessly) to understand the multi-sensory appearance of a rose as a metaphor for the emotional and cognitive notion of love. According to this view, diagrams are schematic icons that are neither particularly strong or weak, nor particularly complex or simple. I hence believe that image, diagram, and metaphor differ in grade rather than in definite quality and therefore form a continuum.
I have earlier suggested that variations of iconicity in all kinds of media can be approximately categorized with the aid of these Peircean types of iconicity and the nature of the representamen and the object; the purpose of such classifications would be to facilitate methodical analysis and to detect vital similarities and differences among media types. From a medial point of view, meaning is always ultimately based on sensory input from a material media product.
However, only two of the exterior senses are cognitively developed to the extent that they can support complex iconicity. Although the other senses should certainly not be excluded from the discussion, it is sufficient to consider only sight and hearing in order to form a lucid model of the basic iconic categories. When the mind processes visual and auditory sensations, this forms cognitive structures that may be perceived as representamens for both other visual and auditory objects and other cognitive structures. Because of our mind’s inclination to grasp also abstract ideas in terms of spatial relations, we are strongly inclined to perceive similarities not only between different kinds of sensory perception, but also between sensory perceptions and cognitive structures. Formal resemblance between sensory perceptions and cognitive import is the ground for iconicity, whether the cognitive operations required to establish it are simple (as in the case of images) or complex (as in the case of metaphors). In other words: iconicity very much depends on spatial thinking, a notion based on research from a range of disciplines, including philosophy, psychology, cognitive theory, and neurology.
Therefore, while meaning in media is always based on initial sensory input, the visual and the auditory must be complemented with the notion of cognitive structure if we are to reach a more complete understanding of how iconic meaning is produced. The chain of semiosis always includes cognitive structures that are drawn into the interpretation at an early stage. Ultimately, cognitive signs tend to produce the most profound types of meaning, which is why sensory aspects are often neglected in the study of literature. However, such inattention offers a warped idea of how literary meaning arises.
I hence distinguish between three main types of representamen: visual material representamens, auditory material representamens, and cognitive representamens. Similarly, three main types of object can be discerned: visual material objects, auditory material objects, and cognitive objects. A visual representamen may iconically represent visual objects, auditory objects, and cognitive objects. A cognitive representamen may represent cognitive objects, visual objects, and so forth. In addition, these types of representation can be understood in terms of metaphor, diagram, and image.
If this conceptual framework is supplemented with the other sense faculties it can be used to analyze iconicity in all kinds of media, including a broad range of objects and phenomena such as depictions in art and photography, ideograms, musical scores, curve charts, computer generated visualizations, sculptures, scientific models, musical meaning, verbal metaphors, onomatopoeia, linguistic synesthesia, allegory, and much more. Visual iconicity in poetry constitutes only a small segment of the many varied types of visual iconicity. Visual iconicity, in turn, is part of iconicity generated by all sensory modes and cognitive structures. On the whole, I believe that iconicity is a vastly underestimated sign category.
In order to illuminate the practical function of visual iconicity also in poetry that may seem to be entirely symbolic, I concluded with an analysis of Sylvia Plath’s “I Am Vertical”, written in 1961. This can now be found in an open access article in Orbis Litterarum, listed below.
The presentation was based on the following publications by Lars Elleström:
2010, “The Modalities of Media: A Model for Understanding Intermedial Relations,” in Media Borders, Multimodality and Intermediality, ed. Lars Elleström (Houndmills: Palgrave Macmillan).
2014, “Material and Mental Representation: Peirce Adapted to the Study of Media and Arts,” in The American Journal of Semiotics 30.
2014, Media Transformation: The Transfer of Media Characteristics Among Media (Houndmills: Palgrave Macmillan).
2015, “Visual, Auditory, and Cognitive Iconicity in Written Literature: The Exemple of Emily Dickinson’s ‘Because I Could not Stop for Death,’” in Iconicity in Language and Literature 14.
2016, “Visual Iconicity in Poetry: Replacing the Notion of ‘Visual Poetry’”, in Orbis Litterarum 71.
2017, “Bridging the Gap Between Image and Metaphor through Cross-modal Iconicity: An Interdisciplinary Model”, in Iconicity in Language and Literature (forthcoming).