Month: May 2022

Posted on

PLAYING WITH SOUND: KAREN COLLINS

Interacting with and listening to game sound

The audiences perception of a sound is dependant on how they listen to it, how one listens can be discussed in terms of Michel Chion’s three modes. The first is casual listening, this is defined as recognising the source of the sound, we instinctively gather information about the world through casual listening. This mode provides the listener with the general characteristics of the objects around them. The second is semantic listening, this is the act of listening and interpreting messages in the sound. This could be seen as not only listening to the words someone speaks, but the tone in their voice. Karen Collins provides the example of the beeping tower in fallout 3, one can casually listen to the beeps and determine that there is a tower in the area and maybe even the distance. One can also listen semantically, providing they understand morse code, and determine a message within the sound. Lastly there is reduced listening, this is focusing in on the auditory qualities of the sound irregardless of the context. As a sound designer we must be conscious that the player can be listening in multiple modes at any time.

Collins goes on to explain the development of these listening modes to accommodate interactive gaming. The musicologist David Huron proposes some additions to Chion’s theory with signal listening, sing along and retentive listening. Signal listening refers to how the player listens in anticipation of a signalling sound. Singalong listening is the act of following along with the sound, an example of this would be karaoke games. Retentive listening is the act of listening with the intention of recalling the information later.

Interactivity with sound in video games is largely dependant not on listening, but the invoking of sound. There is a fluidity to interactivity in game sound, to accommodate for this Collins provides a diagram to separate and identify the different categories of interaction.

Cognitive/psychological interactions are in the centre of the diagram because Collins sees them as a constant phenomenon from which all other interactions take place. This is considered a constant phenomenon by Collins because ‘active reception’ is a form of interactivity (this is informed by Manovich’s reasonings on the misinterpretation of interactivity in computer based media). There is a distinction between two categories of psychological interactions, that which takes place between the player and the game, and that which extends beyond the context of the game itself.

This text was extremely useful as it provides a good basis from which to approach sound design in games. It did stray a lot into more general interactivity in games which I found to be not as focused and valuable but it was still interesting. With a greater understanding of interactivity in games, one can achieve a higher level of it.

Posted on

LEGEND OF ZEDLDA: BREATH OF THE WILD

Meticulous care was put into the foley for the protagonist of this game, as it is a constant sound throughout the game. There is an option in the game to equip ‘stealth armour’, this makes you less evident to enemies. The care put into the sound design can be seen in this element of the game as the designers did not just turn down the volume of the default armour ‘backpacking’ sounds, rather they rerecorded the foley handling the items more carefully. This is not only a more accurate representation of the sound, it also provides a subtle auditory narrative, caution. It’s this care that enriches the sonic world of the game and therefore the immersion of the experience.

Another example of care is the variation in water sounds, different objects have a specific sounds when dropped into water. The sound is dependant on the size of the object, this is also the case with objects on fire, an arrow on fire sounds like a smaller fire than a larger weapon a blaze. This informs the player of how different fires have different uses in the game. When designing the sonic world, it seems one of the most important thing to consider is how objects interact with different elements in that world. These variations in sounds give the objects a weight in reality.

Currently my practice is more focused on the musical side of sound design, and the music in this game is very well thought out. Each village in the game has a theme that reflects the environment, not only that but each has a day and night version of their theme that transition seamlessly into each other. I think that this is a very smart decision on the sound designers part, it makes the environments informationally denser and enriches the players experience of each town. There is new musical information to be discovered based on the amount of time the player stays in the town. This reminds me of a concept I read about in the article ‘Extending Musical Form Outwards in Space and Time: Compositional strategies in sound art and audiovisual installations‘. In the same way installations should be an invitation for perceptual investigation, so should games. As a sound designer for games we must consider the modulating factor of the agency of the mobile visitor, it is a modulating factor for our creations.

I think one of the most effective use of sound design in this game is the interweaving of music and action in combat situations. The game is designed to play combat themes when the player gets within a certain distance of an enemy, there are many variations of this theme depending on the difficulty of the opponent. The most interesting part of this design, in my opinion, is the fact that musical accents are added to the theme when a player lands a hit on an opponent. These accents are also modulated by the strength of the hits, the stronger the hit, the louder the musical accent. These accents are added to the beat of the bar directly after the hit. This piece of innovation is genius in my opinion, as it blurs the line between musical score and action. The musical score is there as a way to influence the player, to amp them up in combat. However the player is influencing the score while engaging in the combat, resulting in a kind of feedback loop of exhilaration.

There are multiple sound design techniques implemented in this game that I wish to engage in, and researching them has given me a better appreciation of how sound design can provide a lot of information to the player.

Posted on

ELEMENT 2

Overall Reflection

When I started this line of enquiry I had a pretty clear idea that diegetic underscoring would be a more effective means of scoring than the traditional method, however through research and experimentation I have become more familiar with the intricacies and difficulties of the method.

My first difficulty is that the context within which this method can be used seems to be fairly niche, not every cinematic scene or situation lends itself to diegetic scoring. It’s completely dependant on the emotional narrative of a scene, because a diegetic composer is limited in their approach. If a scene requires an intense composition but is set somewhere with minimal sound, the composer will find their job very difficult. This could result in very creative uses of foley or ambience and doesn’t negate the possibility of great work, I am only acknowledging that this is a difficulty specific to the practice.

The second point id like to make is that using diegetic elements in a musical score seems to be more common than I first thought, there are multiple composers I have researched that engage in this practice as a diegetic anchorage. I think my enquiry was less of an overview of the practice, as it was an exploration of the spectrum of audiovisual correspondence within the practice. Through my experimentation and creative piece I discovered that the approach to diegetic underscoring boils down to abstraction, the degree of abstraction one wants to engage in. The higher the abstraction the higher the emotional intensity (in the majority of cases), however the abstraction detracts from the audiovisual correspondence and therefore the conceivable diegesis of the sound.

The last problem I encountered was the realisation of the piece, the creation of these two audio works was an invaluable process of experimentation and learning, but I wouldn’t say that they reflect the method in the best possible way. I think the first issue was the lack of a film scene to work from, if I were to have a scene I would be able to map out and choreograph the intensity and synchronisation to match the intensity of the scene. Unfortunately I had to conduct the recording as more of an experiment, investigating the transition from one state to another. Chaos and synchronisation. Another problem I had was the lack of participants, the recording would have been more effective had there been more people, and ideally dancers as they would have a better idea of how to use their body. I found the recording difficult to mix into a scene, to me the recording of the eight participants and the atmosphere sound separate. To remedy this I would, as I’ve said, have more participants in a larger space, and perhaps record more footsteps in the same space to mix into the ambience.

I think my piece works well as a middle ground between ambience and foley, and as such is an excellent avenue for subtle manipulation of the audience. The recordings may not be perfect due to the issues listed above but the process worked as proof of my research. If this method were to be used in a production with a decent budget, I as the sound designer would be working from a film scene, be able to increase the level of choreography tailoring it to the narrative, hire more participants who have a dancing background, and record with consideration to the implementation of the sound in the diegetic environment (If I had the time, patience, money and opportunity I would create a convolution reverb at the location of shooting).

Minimal abstraction with dialogue-

Higher abstraction no dialogue-

Posted on

LUDWIG GORANSSON

Tenet

Ludwig Goransson is a Swedish record producer, conductor and film composer. He has worked on productions such as Creed (2015), Black Panther (2018) and The Mandalorian (2019-2022). I will be focusing on his work on the 2020 Christopher Nolan film ‘Tenet’.

One of the most interesting techniques Goransson used for this film was reverse scoring. Ludwig scored a brass band piece, then reversed the score, recorded an orchestra performing that piece, then reversed that recording. This resulted in a very interesting sound, you can hear that the composition is intended that way, not reversed. However all elements within the overall composition are reversed, which is a very pronounced as brass instruments tend to have a distinctive envelope. This score was not only sonically interesting but also thematically relevant as the film is about reversing entropy.

‘The recording is playing backwards but the theme is playing forwards’

This reminds me of the letter from a composer in Schaeffer’s in search of concrete music, drastic manipulation while maintaining the overall form/structure of the music.

For the ‘trucks in place’ score, Gorranson utilises the sound of a truck engine for the low frequency rumble. The recording is pitched down and manipulated, but retains the essence of the truck. This is a version of what my research is about, its a diegetic anchor in the music. I think that the music for a film scene is more impactful and interesting for the listener when the instruments are informed by the screen/context.

‘You’re in this world with these trucks and firetrucks and buses, and the music you’re hearing is also coming from that world. Its not too on the nose … manipulated sounds from the world you’re actually in’

Goransson utilised a similar technique for the film Creed, he recorded a boxer doing their workout and used the recordings as musical elements for the score. These recordings also acted as inspiration informing the composition. The sound of a train was also used in the score as a train plays a prevalent part in the film, the rhythms of the tracks are manipulated and weaved into the score.

‘The movie was so powerful so you didn’t need a traditional score, you didn’t need melodies to tall the audience what he was going through, everything is just glued on the screen. So you just want the music to kind of highlight and heighten the tension and the uncomfortableness that you’re watching on the screen.’

Posted on

ELEMENT 2

Composition

I decided to create two compositions with varying levels of interference/manipulation.

The first is minimal manipulation, I used the original recording of the footsteps session and added a train station ambience. I also decided to use some audio of intense cinematic dialogue, I did this as a means of distracting the listener, giving them something to follow. This is because the footsteps are meant to be a support for a narrative, rather than the focus of the scene. I wanted the footsteps to have a certain level of subtlety to them, so as not to take away from the story. Unfortunately the recording demands more attention than I had anticipated, I think that this is due to the transition between normal and uniform footsteps was too short. Ideally I would like a more gradual transition to give the audience time to acclimatise to the sound, the subtler the better. I would say that this piece is a success, it conveys what I wanted it to convey, its a completely diegetic underscore that supports the narrative of the scene.

For the second version of the piece I took a more abstracted approach to the composition, allowing for more manipulation and more remote audiovisual correspondence. The first challenge I faced was the variations in tempo, the participants were not perfect in their steps and the synchronisation moves in and out of the 115 BPM I had set with a visual metronome. I had to find an instance of consistent tempo, edit it slightly and duplicate the clip. This is the first step away from audiovisual correspondence as the ambient sound has now been sampled. I then imported a field recording I had made a few years ago, the recording was of some construction work being conducted near my house. The sound I decided to loop was of what I think is a circular saw cutting stone. An uncomfortable sound to support an uncomfortable scene. I manipulated this sound with a tremolo, this gave the sound a rhythmic quality blending nicely with the steps. I was using Ludwig Gorranson’s compositions for the 2020 Nolan film ‘ Tenet’ as a reference, I wanted to create a similar palette. I also applied a tremolo to a field recording of wind. To emphasise the first beat of the bar I used two elements, a reversed clap and a sped up recording of an egg shaker increasing in tempo. Both of these sounds are very high in frequency so they cut through the constant low frequencies from the contact mics. Lastly to add some melody I manipulated a recording of a train engine. This choice was informed by the writings of Pierre Schaeffer on sound retaining meaning (specifically the train and the foundry).

This more musical approach was interesting, I felt I could achieve a more dense level of emotional information but the manipulations detracted from the audiovisual correspondence. The main problem I had with my process was that I had decided to use Protools as a workstation to compose, this resulted in many problems, from instruments not working to audio files not being able to be imported, mainly just a slower and less efficient process of creation.

Posted on

VISITING PRACTITIONERS

KMRU

KMRU (Joseph Kamaru) is an ambient music artist who has a focus on field recording. He is drawn to meditative and calming music, possibly as a result of his upbringing in the noisy streets of Nairobi. He describes it as a means to ‘slow things down.’

In high school, Joseph moved from the city to a more rural area. This drastic change of sonic environment caused a sensitivity to his surroundings and a focus on his aims. Kamaru started to conduct field recordings, collecting material to use in his music production. He emphasises minimalism as a means of inspiration, the limitation forces you to make the best work with what you have. There is also an emphasis on being open to whatever happens, improvising on synthesisers while not restricting the outcome.

Kamaru is currently doing a three year master’s program in the Berlin university of arts. The transition from Nairobi to Berlin was difficult as the sonic landscapes are so different, Kamaru started to wonder if his sound would change. Joseph’s intention and understanding of field recording developed as he went deeper into the practice. His relationship to the practice was strictly functional, he valued the recording itself, for its sonic qualities and utility for music. Now, Kamaru has started to consider the location as an important aspect of the practice.

‘I feel that sound itself is the best way to learn and understand our surroundings‘

Kamaru has been making an effort to remove the act of recording occasionally, just to sit and listen as a meditative activity. Sometimes he will bring a notebook and pen to describe his surroundings, this seems to be a method of becoming more perceptually literate. Experience is as important as the act of recording. Occasionally he will draw what he hears as a way to document experience.

Kamaru’s ambient music philosophy is to use elements of recordings in extended ways to create new textures/melodies. The aim of his method is to blend music with field recording to create a sense of insecurity in the listener as to whether music is happening, the field recordings also provide a narrative behind the music. The listener has something to connect to enhancing their experience.

I find KMRU’s work inspiring as its a practice I’m unfamiliar with, I don’t make ambient music and I only rarely conduct field recordings. His idea of giving the listener something more to connect to and giving instruments a narrative is very interesting. I believe this philosophy applies to my interest diegetic underscore, the recordings you compose with carry a narrative that is evident even after manipulation.

Posted on

ELEMENT 2

Recording session

First thing I did was mark out the sections for the spatial part of the piece. Chair for scale.
Then I installed the recording equipment. It consisted of a 3Dio Binaural mic and two contact mics running as two mono channels.
3Dio Binaural microphone
Contact Microphone
Checking levels

The recording session went well, there was not as many people as I had hoped but the experiment still worked. Eight people attended the experiment, this resulted in a decent ‘crowd’ sound. The first recording went well, the lack of definite tempo and lack of practice made the synchronisation gradual (which is what I was aiming for). I set a tempo of 115bpm for the next recording, this is because I believed it to be a good tempo for composition if I want to layer more musical elements on it. The synchronisation was less gradual with each recording as the participants became more familiar with the action, which I should’ve anticipated but I believe with more participants the synchronisation would take more time. The one thing I forgot to mention to the participants until the last recording was to increase the volume of the footsteps after the synchronisation had occurred, I assumed that this would happen anyway but it didn’t.

Im glad I decided to use contact microphones as I believe those recordings will come in very useful for creative manipulation. My initial idea is to bring the contact mic recording of the synchronisation section up in the LFE, preceding the binaural synchronisation. A subtle foreshadowing of the incoming intensity.

Posted on

MA COLLABORATION

What remains of Edith Finch

In our first meeting the MA group told us that their game was influenced by this 2017 game ‘What remains of Edith Finch’. This is a first person magical realism game in which the player gradually uncovers the history of the Finch family. The player explores the family home, uncovering clues to generations of the family, the story of their deaths, and the curse that haunts them. The object of this game is not to win, but to spend time immersing yourself in the stories of each Finch’s life.

The player must discover hidden passageways to enter each room, the whole house having an etherial quality, the game has been described as reminiscent of a pop up book. The player is transported from one story to another, each with their own palette. The stories of their deaths are told in very creative and thought provoking ways, including allegories from the perspective of animals, and text from the notes found in the house integrated in the scenery of the story.

Ian Dallas (the game’s head designer) talks about the sound being influenced by films such as Donnie Darko and Under the Skin. He says he listened to the soundtracks as a means to get into a dreaming headspace to conceptualise the game.

I can definitely see how the MA group’s ‘Reverse’ is influenced by this story driven style of game and the etherial quality of Edith Finch. I think this otherworldly palette is a deep well for creative licence and should be very fun to play around with. I am focused primarily on music in my practice, so this will be most of what I do, but I think this otherworldliness is achievable through reverb on the foley and atmospheres. Due to the game’s dream like aesthetic I hope to achieve a level of disorientation or weightlessness in parts of the sound design, our team has an opportunity to contrast this dreamscape with the weighted dreary world of the office. Depending on how the game turns out visually, this could be a very interesting and effective juxtaposition of worlds.

Posted on

ELEMENT 2

‘Planning recording session’

‘Black Box’ Room at LCC

I have decided to use footsteps as a means of diegetic underscore as they are present in a large quantity of film and television, they also offer an opportunity to compose with rhythm, chaos and synchronisation.

The purpose of this entire practice/experiment is to find a less intrusive way of influencing an audience, maintaining their suspension of disbelief. Footsteps permeate everyday life, we hear them constantly, this makes them an excellent candidate for manipulation. Through manipulation/choreography, this sound can go from its natural chaotic state, to a synchronised rhythm. provided this synchronisation is gradual enough, the audience won’t notice the transition from close to remote audiovisual correspondence. Technically the audiovisual correspondence or synchronous integrity of the sound cannot be destroyed, as the sound source could be constantly present in a scene (People in a train station), and it is not impossible that people’s footsteps would synchronise in that narrative world, unlikely but not impossible.

Fig.1

I think it is necessary to use 3 types of microphones in the recording session to maximise utility of the sound. Firstly a binaural microphone to capture a realistic spatial recording of the space when this synchronisation happens, this will especially emphasise the immersion and experiencial connection between character and audience. Secondly a condenser microphone to capture an overall perspective of the room, I think this is necessary as I believe it’ll be easier to use in a mix. Thirdly a contact mic (maybe more than one) to capture the impact of the feet on the floor. These lower frequencies could be useful in a 5.1 mix, for the LFE to drive more impact into the mix. I’m considering using an ambisonic microphone for the possibility of implementation into VR.

I will be conducting the piece in two ways with varying levels of interference/choreography. The first will be minimal choreography (Fig. 2), I will ask the participants to walk normally (as if they were walking to catch a train, maybe even ask to avoid synchronisation), then after 40 seconds ask the participants to synchronise. This will result in unpredictable results as it’s impossible to know how long it will take them to synchronise, but with result in a very organic transition.

Fig.2

The second level of choreography (Fig.3) is separating the people into groups (for groups see Fig.1), after 30 seconds I will signal group 3 to synchronise, then at 45 I will signal group 2 to synchronise, then at 60 seconds group 1.

Fig.3

An important aspect of this recording session is the material of the floor, obviously carpet would be too quiet, but the material must also match the materials present in film scenes. For a train station I think the ‘Black Box’ room is the best option, it has a slight rubber texture to it (Fig. 4). It’s a little quieter/softer than I would have liked, but it is usable.

Footsteps in Black Box room recorded by holding iPhone 12 while walking
Footsteps in Black Box room recorded by placing iPhone 12 on chair (Fig. 1)

Posted on

ADAPTIVE AUDIO

Video game sound designers have a unique task in the world of sound design. In a film, the sound designer knows exactly what will happen, but in a game the designer must account for the player’s agency. The world will change depending on the player’s interaction with it. This is ‘adaptive audio’, audio that adapts to player input. An example of adaptive audio in game design is footsteps, they must first be tied to the event of walking (then the player inputs the signal to walk), the audio must then be matched to the walking animation. The necessary work to achieve audiovisual correspondence in games is far more than film. There is another degree of adaptivity beyond this, adapting to environment. The footsteps must change sound depending on where the player moves their character. This is especially important when there is an informative element to it, for instance if ice is slippery there should be an auditory cue for the player.

Another degree of adaptivity is how the sounds we implement interact with the environment in the game. Inaccurate reverb will bring the player out of the game, ruining the immersion. As sound designers our job is to immerse the player, and to avoid distracting their attention from where it ought to be.

To solve these problems we must include variation within the sounds triggered by the event. This can be done though designating certain reverbs to certain zones within the game. We can also implement gradual gradual transitions that is dependant on movement thought environments, this is usually tied to music transitions from the different locations. The most effective utilisation of adaptive audio, in my opinion, is when designers pitch sound effects to key of the background music. This results in a merger between diegetic interactive sound and the non-diegetic sonic/melodic world of the game. I hope to have the opportunity to explore implementing these parameters in FMOD.