Beyond the Visual: Applying Cinematic Sound Design to Digital Environment

Posted: October 3, 2011 by Shuning Li in 2011, interaction, sound

Student Name& ID: Shuning Li (11261005)

Beyond the Visual: Applying Cinematic Sound Design to Digital Environment


In the early 20th century, sound was integrated into filmmaking, which was used to support motion pictures and contribute to the emotional and narrative design of film projects. In the later of the century, both of the discipline and technology of the sound design flourished with the rapidly development of the modern world, sound became an integral part of the media practice, which played substantial roles in filmmaking[1].

However, as the media entered into the digital age, while the mechanisms of engagement and immersion was studied extensively in terms of story and visual images, sound design enjoyed less attention in theory studies, much less the considerations of applying sound to digital interactive environment.

According to Mark Ward and Linda Leung, interactive media industry should learn the best of what the last 100 years of sound design practice and apply the techniques to the digital environment. They leveraged four important functions of sound relevant to both film and interactive media which filmmakers could emphasis and work on to created effective sound tracks for digital interactive media, that are emotional truth, point of view, storytelling and physical experience. They suggested the aims of sound design for digital interactive media is to elicit the mood and emotion from its audience, to draw users into its world as ‘we have eyelids but not earlids’, that is, we can block out the visual merely by closing eye, but can barely escape the aural (Leung, 2008).

A Brief History of Film Sound Design

The history and the art of sound is not as sophisticated as for the moving image, and yet sound informs and anchors the moving image.

Early cinema is perceived as 'silent'Early cinema is perceived as ‘silent’

Early cinema is commonly perceived as ‘silent’ (as it was a visual medium only) because no sound information was printed upon the filmstrip. As the media industry enters into the digital era and the notion of film is gradually assumed as the digital convergence between image, audio, and online experience, among others, and thus the practice of multi-media. The multi-media practice, especially electronic media broke the tradition of “virtual reality” and brought the new concept of “augmented reality”. Now the audience can really “break through” the screen, with no more mediation and no more separation to see, feel and touch “the myth of total cinema,” as André Bazin put it (Elsaesser, 2006), a renowned and influential French film criticand film theorist.

5.1- channel digital surround sound (DSS) in cinema5.1- channel digital surround sound (DSS) in cinema

The language and vocabulary of sound has largely expanded with the development of modern technology and the digital media, which can be seen from various new techniques and devices for sound design and production, such as, the 5.1- channel digital surround sound (DSS)[2] in cinema, Pro Tools digital audio workstation software and Musical Instrument Digital Interface (MIDI). A specialized craft of four major aural categories of the sound design for film also indicates the evolution of sound media, which includes music, ambient sound, sound effects and dialogue. Dialogues and narrative sound effects are mostly used to establish the narrative structure of stories, while music and ambient sound are used more and more often in building atmosphere and locating film style.

One of the early innovations in the use of sound in digital interactive environments for example, was made by Apple Computer, which deployed personal computers into the marketplace equipped with sound-cards, as well as plug-in or build-in microphones allowing user-generated material to be recorded and manipulated, or linked to other documents. Most importantly, Apple’s Macintosh user interface was conceived of in visual and aural terms, embedding William Gaver’s work in auditory icons (1986) into its philosophical core (Leung, 2008).

Roles of Sound

Film is described as an audio-visual (AV) medium. In the term ‘AV’, audio and visual is linked with each other, which suggests that sound and picture together can become greater than the sum of the parts. The term is also assumed as the audio precedes the visual, that is, not only is sound integral, it is prioritized.


Sound mainly plays narrative, subliminal and grammatical roles in film, according to Tomlinson Holman. Sound may tell the story directly, or it may be integrated along with pictures into a complete whole. For instance, Dialog and narrative sound effects, two important techniques for storytelling, which are used to write into the script to notify what is happening and what corresponding action actors should take at the moment.

One Missed CallTelephone Ringing Sound in Horror Film ‘One Missed Call’

Sound also has a subliminal role, working on its audience subconsciously, which in simplicity, is to build atmospheres, deliver emotions and enhance moods of stories. For example, intense and unpleasant sound experience could triggers negative emotions lasting hours or even days, which used by filmmakers in horror and black film genre.

In terms of the grammatical role in filmmaking, sound provides “a form of continuity or connective tissue for films” (Holman, 2002). When a picture is cut, but the scene is not shifted, sound usually remains constant before and after the shot in that case, to indicate audience that although the point of view may have changed, the scene is not shifted, and therefore we are still in the same space as before. In particular, ambient sound is used most to present the continuity of a scene. Looking through the major roles of sound in filmmaking, audience in most cases wouldn’t even realize this natural and seemingly effortless sound design, but it is absolutely indispensable to the expressions of films. Images wouldn’t make a complete story without the combination of sound.

Applying Sound Design to the Digital Environment

Mark and Linda’s study highlighted four functions of cinematic sound design, which the digital media industry could emphasize on in digital interactive media experiences. The primary function of sound in both films and interactive media work is to construct and communicate emotions. Emotion is the soul of a media work, which decides all the information in the project and guide the whole production process. Sound design is essential and effective to set the emotion and mood in a media project, which in turn, orient audience to receive and interpret the visual information.

In-car GPSSelecting voice for spoken navigation instructions on an in-car GPS system

The second function is to establish point-of-view, or in this case point of audition. In film, sound steers audience’s attention through visual information, by promoting important details. In online experience, sound can represent the ‘voice’ of an organization, or an acoustic ‘buzz’ of a company. Some other digital media allow sonic customization, whereby a user can self- select their preference and express their own point-of- view, such as choosing either male or female voice for spoken navigation instructions on an in-car GPS system.

Online game requires sound qualityOnline game requires lifelike sound effects

Sound also functions to structuring storytelling. Sound connects and “dubs” the visual elements in film, thereby ascribing meaning and legitimizing it as a trustful version of the real world. In digital interactive experiences, some online game for instance, lifelike sound effects combined with motion pictures is one of the selling points, which literally create a virtual world for online players.

The final function of sound is to create physical experience. Sound contributes to the overall multisensory experience in cinema as well as in interactive media. ‘Hearing is a way of touching at a distance’ (Leung cited in Schafer, 2005), that is, hearing is a form of touch as it has a synaesthetic quality. Sound can “have a synaesthetic role in interactive media, where sound is felt as much as it is heard, as can be seen in video games where explosive sounds are usually accompanied by vibrations in the console. That the senses of hearing and touch are so closely aligned provides a compelling argument for sound design to be a critical component of interactive media development” (Leung, 2008).


Film is an interaction of sound, visual and story, and filmmaking is a progress of combining these elements in ways that make the impact of the resulting experience larger than the sum of its parts. As a historical and fertile medium, sound impacts on the way films are made as much as image. With the rapid development of modern technology and increasing influence of digital media, the world enters into the digital era, therefore, to create effective sound tracks for the digital interactive environment become of great importance. Mark Ward and Linda Leung suggest that the digital media practitioners could learn from cinematic sound design experiences, and apply them to digital interactive practice.


Elsaesser, T. (2006). Early Film History and Multi-Media- An Archaeology of Possible Futures. In W. H. Keenan, New Media, Old Media: a history and theory reader (pp. 13-35). New York: Routledge.

Holman, T. (2002). Sound for Film and Television. USA: Butterworth-Heinemann.

Kerins, M. (2008). Beyond Dolby(stereo): Cinema in the Digital Sound Age. USA: Indiana University Press.

Leung, L. (2008). Digital Experience Design. UK: Intellect Books.

Nyre, L. (2008). Sound Media: From Live Journalism to Music Recording. London: Routledge.

[1] The term refers to its universal meaning in this article, that is, the general range of activities required to make a film, video, or television program.

[2] Digital “5.1” Sound- the “5”referring to the configuration’s five full-range channels and the “.1” to its bass-frequencies- only low-frequency effects (LFE) channel. The proposal of the adoption of 5.1-channel digital surround sound was brought up by Tomlinson Holman in the late 1980s (Kerins, 2008).

  1. amynasha says:

    It is an informed report, I feel that it is useful as a film-maker. But ‘AV’ term is a bit lol, because as Chinese we all know it what usually means. Super lol…

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s