MA COLLABORATION

Integrating sounds

Now that we received a video of the game we’re able to start integrating the sounds, this has turned out to be an interesting process. Due to the circumstances of our collaboration everyone in the group has been creating the different elements of the game separately, working from the diagrams and notes provided by the MA group. Although not the best method of collaboration, this has proved to be useful due to the depth of sound required for the game. This way we were able to split the workload and cover a wider spectrum of the game, the draw back to this compromise was a lacking of coherency. There was a general coherency in sound, I believe there is a standardised ‘mini-game sound’ quality. However I feel given more time there could’ve been more coherency. There are three tracks used for the game provided by Pedro, Max and myself. Pedro’s track reflects the style of the mini games, light hearted, synthesiser heavy, playful music. Max contributed a track that provided an appropriate musical backing to the dreamscape of the 2nd level with synthesised game like elements. I feel as though my contribution is the only track that doesn’t have a game aesthetic, I went for a merger between industry and nostalgia, an accurate reflection of the theme of the game, but an aesthetic anomaly. This difficulty was a result of our disjointed collaboration, to remedy this I would have composed with instruments that had a game type timbre and integrated that into the convolution.

Working from the video proposed another level of difficulty as we had recorded and organised our foley as if it would be inputed in FMOD or Unity, by this I mean as single steps and actions. This meant when I came to input these sounds into the video I had to import and arrange individual steps and clicks, a tedious and time consuming process. If I were to undergo this process again from the start with the understanding I have now having seen the full game, I would try to merge game sound with office sound to re-enforce the narrative of the game. This would either mean gradually integrating game sounds into the foley of the office, or integrating office sounds into the dreamscape the connect these two worlds.

We have received the game project file, while I am working on this video to give an overall idea of our sound collaboration, I am going to try and get someone else to attempt to integrate some of our sounds for part of the game using Unity or FMOD. It would be good to integrate our sounds in this way as I believe they would work together better in an immersive and interactive context.

MA COLLABORATION

Merging audio using convolution reverb with custom impulse response

Using logic’s space designer I was able to customise an impulse response as a means of merging audio files, as mentioned by Martin Stig Anderson in his lecture.

I wanted to integrate music into the game in an immersive way, so as to emotionally influence the player while maintaining suspension of disbelief. By intwining the melody with something in the game I was able to diegetically anchor the music. The audio I chose to use for this was the sound of a foundry:

Foundry Audio

I chose this because I believed it shared a contextual/timbral quality with the kaleidoscopic room from the third level, this level being a good summary of the theme and feel of the game. Disorientating, curious and otherworldly.

The first experiment I conducted was applying a convolution reverb to the music, the convolution reverb had the sound of the foundry as it’s impulse response. The result was a strange distortion of the music, into a kind of soundscape. It lost its form and became a kind of fluid representation of the composition the was influenced by the foundry. I would ideally like this influence to be more prominent but I couldn’t find any way to make this the case, even after a lot of tweaking.

Music through room convolution

I wanted to try the process the opposite way to see if the effect was more successful. I found it hard to understand what each element of modulation in the convolution reverb did, but though general experimentation I managed to gain a basic understanding. The most important thing I didn’t manage to understand was how to control the degree of abstraction away from the original form of the audio. This experimentation had a lot less form than the previous one, the music lost its melodic variation and became one tone. Almost like a summary of the composition layered on the foundry.

Foundry through Music Convolution

I found a compromise by using the ‘music through room convolution’ and layering the original foundry audio with a convolution reverb that had the same impulse response (although the wet and dry levels were adjusted to be less drastic). The idea of this was to have an ‘accurate’ reverb under the original recording, that could blend with the musical reverb. It provides an extra tether to reality.

Music trough room convolution with layered foundry

This experiment was a success, it is a diegetically anchored sound scape composition. I think for this method to be more successful (by that I mean retain more of the diegetic form), I would have to understand the convolution merging technique better and possibly compose the music with the exact same timbre and similar frequencies as the diegetic sound. The composition reflects the game’s themes of nostalgia and self reflection, whilst not taking the player out of the experience, it fits the disorienting world of the game.

PLAYING WITH SOUND: KAREN COLLINS

Interacting with and listening to game sound

The audiences perception of a sound is dependant on how they listen to it, how one listens can be discussed in terms of Michel Chion’s three modes. The first is casual listening, this is defined as recognising the source of the sound, we instinctively gather information about the world through casual listening. This mode provides the listener with the general characteristics of the objects around them. The second is semantic listening, this is the act of listening and interpreting messages in the sound. This could be seen as not only listening to the words someone speaks, but the tone in their voice. Karen Collins provides the example of the beeping tower in fallout 3, one can casually listen to the beeps and determine that there is a tower in the area and maybe even the distance. One can also listen semantically, providing they understand morse code, and determine a message within the sound. Lastly there is reduced listening, this is focusing in on the auditory qualities of the sound irregardless of the context. As a sound designer we must be conscious that the player can be listening in multiple modes at any time.

Collins goes on to explain the development of these listening modes to accommodate interactive gaming. The musicologist David Huron proposes some additions to Chion’s theory with signal listening, sing along and retentive listening. Signal listening refers to how the player listens in anticipation of a signalling sound. Singalong listening is the act of following along with the sound, an example of this would be karaoke games. Retentive listening is the act of listening with the intention of recalling the information later.

Interactivity with sound in video games is largely dependant not on listening, but the invoking of sound. There is a fluidity to interactivity in game sound, to accommodate for this Collins provides a diagram to separate and identify the different categories of interaction.

Cognitive/psychological interactions are in the centre of the diagram because Collins sees them as a constant phenomenon from which all other interactions take place. This is considered a constant phenomenon by Collins because ‘active reception’ is a form of interactivity (this is informed by Manovich’s reasonings on the misinterpretation of interactivity in computer based media). There is a distinction between two categories of psychological interactions, that which takes place between the player and the game, and that which extends beyond the context of the game itself.

This text was extremely useful as it provides a good basis from which to approach sound design in games. It did stray a lot into more general interactivity in games which I found to be not as focused and valuable but it was still interesting. With a greater understanding of interactivity in games, one can achieve a higher level of it.

LEGEND OF ZEDLDA: BREATH OF THE WILD

Meticulous care was put into the foley for the protagonist of this game, as it is a constant sound throughout the game. There is an option in the game to equip ‘stealth armour’, this makes you less evident to enemies. The care put into the sound design can be seen in this element of the game as the designers did not just turn down the volume of the default armour ‘backpacking’ sounds, rather they rerecorded the foley handling the items more carefully. This is not only a more accurate representation of the sound, it also provides a subtle auditory narrative, caution. It’s this care that enriches the sonic world of the game and therefore the immersion of the experience.

Another example of care is the variation in water sounds, different objects have a specific sounds when dropped into water. The sound is dependant on the size of the object, this is also the case with objects on fire, an arrow on fire sounds like a smaller fire than a larger weapon a blaze. This informs the player of how different fires have different uses in the game. When designing the sonic world, it seems one of the most important thing to consider is how objects interact with different elements in that world. These variations in sounds give the objects a weight in reality.

Currently my practice is more focused on the musical side of sound design, and the music in this game is very well thought out. Each village in the game has a theme that reflects the environment, not only that but each has a day and night version of their theme that transition seamlessly into each other. I think that this is a very smart decision on the sound designers part, it makes the environments informationally denser and enriches the players experience of each town. There is new musical information to be discovered based on the amount of time the player stays in the town. This reminds me of a concept I read about in the article ‘Extending Musical Form Outwards in Space and Time: Compositional strategies in sound art and audiovisual installations‘. In the same way installations should be an invitation for perceptual investigation, so should games. As a sound designer for games we must consider the modulating factor of the agency of the mobile visitor, it is a modulating factor for our creations.

I think one of the most effective use of sound design in this game is the interweaving of music and action in combat situations. The game is designed to play combat themes when the player gets within a certain distance of an enemy, there are many variations of this theme depending on the difficulty of the opponent. The most interesting part of this design, in my opinion, is the fact that musical accents are added to the theme when a player lands a hit on an opponent. These accents are also modulated by the strength of the hits, the stronger the hit, the louder the musical accent. These accents are added to the beat of the bar directly after the hit. This piece of innovation is genius in my opinion, as it blurs the line between musical score and action. The musical score is there as a way to influence the player, to amp them up in combat. However the player is influencing the score while engaging in the combat, resulting in a kind of feedback loop of exhilaration.

There are multiple sound design techniques implemented in this game that I wish to engage in, and researching them has given me a better appreciation of how sound design can provide a lot of information to the player.

ELEMENT 2

Overall Reflection

When I started this line of enquiry I had a pretty clear idea that diegetic underscoring would be a more effective means of scoring than the traditional method, however through research and experimentation I have become more familiar with the intricacies and difficulties of the method.

My first difficulty is that the context within which this method can be used seems to be fairly niche, not every cinematic scene or situation lends itself to diegetic scoring. It’s completely dependant on the emotional narrative of a scene, because a diegetic composer is limited in their approach. If a scene requires an intense composition but is set somewhere with minimal sound, the composer will find their job very difficult. This could result in very creative uses of foley or ambience and doesn’t negate the possibility of great work, I am only acknowledging that this is a difficulty specific to the practice.

The second point id like to make is that using diegetic elements in a musical score seems to be more common than I first thought, there are multiple composers I have researched that engage in this practice as a diegetic anchorage. I think my enquiry was less of an overview of the practice, as it was an exploration of the spectrum of audiovisual correspondence within the practice. Through my experimentation and creative piece I discovered that the approach to diegetic underscoring boils down to abstraction, the degree of abstraction one wants to engage in. The higher the abstraction the higher the emotional intensity (in the majority of cases), however the abstraction detracts from the audiovisual correspondence and therefore the conceivable diegesis of the sound.

The last problem I encountered was the realisation of the piece, the creation of these two audio works was an invaluable process of experimentation and learning, but I wouldn’t say that they reflect the method in the best possible way. I think the first issue was the lack of a film scene to work from, if I were to have a scene I would be able to map out and choreograph the intensity and synchronisation to match the intensity of the scene. Unfortunately I had to conduct the recording as more of an experiment, investigating the transition from one state to another. Chaos and synchronisation. Another problem I had was the lack of participants, the recording would have been more effective had there been more people, and ideally dancers as they would have a better idea of how to use their body. I found the recording difficult to mix into a scene, to me the recording of the eight participants and the atmosphere sound separate. To remedy this I would, as I’ve said, have more participants in a larger space, and perhaps record more footsteps in the same space to mix into the ambience.

I think my piece works well as a middle ground between ambience and foley, and as such is an excellent avenue for subtle manipulation of the audience. The recordings may not be perfect due to the issues listed above but the process worked as proof of my research. If this method were to be used in a production with a decent budget, I as the sound designer would be working from a film scene, be able to increase the level of choreography tailoring it to the narrative, hire more participants who have a dancing background, and record with consideration to the implementation of the sound in the diegetic environment (If I had the time, patience, money and opportunity I would create a convolution reverb at the location of shooting).

Minimal abstraction with dialogue-

Higher abstraction no dialogue-

LUDWIG GORANSSON

Tenet

Ludwig Goransson is a Swedish record producer, conductor and film composer. He has worked on productions such as Creed (2015), Black Panther (2018) and The Mandalorian (2019-2022). I will be focusing on his work on the 2020 Christopher Nolan film ‘Tenet’.

One of the most interesting techniques Goransson used for this film was reverse scoring. Ludwig scored a brass band piece, then reversed the score, recorded an orchestra performing that piece, then reversed that recording. This resulted in a very interesting sound, you can hear that the composition is intended that way, not reversed. However all elements within the overall composition are reversed, which is a very pronounced as brass instruments tend to have a distinctive envelope. This score was not only sonically interesting but also thematically relevant as the film is about reversing entropy.

‘The recording is playing backwards but the theme is playing forwards’

This reminds me of the letter from a composer in Schaeffer’s in search of concrete music, drastic manipulation while maintaining the overall form/structure of the music.

For the ‘trucks in place’ score, Gorranson utilises the sound of a truck engine for the low frequency rumble. The recording is pitched down and manipulated, but retains the essence of the truck. This is a version of what my research is about, its a diegetic anchor in the music. I think that the music for a film scene is more impactful and interesting for the listener when the instruments are informed by the screen/context.

‘You’re in this world with these trucks and firetrucks and buses, and the music you’re hearing is also coming from that world. Its not too on the nose … manipulated sounds from the world you’re actually in’

Goransson utilised a similar technique for the film Creed, he recorded a boxer doing their workout and used the recordings as musical elements for the score. These recordings also acted as inspiration informing the composition. The sound of a train was also used in the score as a train plays a prevalent part in the film, the rhythms of the tracks are manipulated and weaved into the score.

‘The movie was so powerful so you didn’t need a traditional score, you didn’t need melodies to tall the audience what he was going through, everything is just glued on the screen. So you just want the music to kind of highlight and heighten the tension and the uncomfortableness that you’re watching on the screen.’

ELEMENT 2

Composition

I decided to create two compositions with varying levels of interference/manipulation.

The first is minimal manipulation, I used the original recording of the footsteps session and added a train station ambience. I also decided to use some audio of intense cinematic dialogue, I did this as a means of distracting the listener, giving them something to follow. This is because the footsteps are meant to be a support for a narrative, rather than the focus of the scene. I wanted the footsteps to have a certain level of subtlety to them, so as not to take away from the story. Unfortunately the recording demands more attention than I had anticipated, I think that this is due to the transition between normal and uniform footsteps was too short. Ideally I would like a more gradual transition to give the audience time to acclimatise to the sound, the subtler the better. I would say that this piece is a success, it conveys what I wanted it to convey, its a completely diegetic underscore that supports the narrative of the scene.

For the second version of the piece I took a more abstracted approach to the composition, allowing for more manipulation and more remote audiovisual correspondence. The first challenge I faced was the variations in tempo, the participants were not perfect in their steps and the synchronisation moves in and out of the 115 BPM I had set with a visual metronome. I had to find an instance of consistent tempo, edit it slightly and duplicate the clip. This is the first step away from audiovisual correspondence as the ambient sound has now been sampled. I then imported a field recording I had made a few years ago, the recording was of some construction work being conducted near my house. The sound I decided to loop was of what I think is a circular saw cutting stone. An uncomfortable sound to support an uncomfortable scene. I manipulated this sound with a tremolo, this gave the sound a rhythmic quality blending nicely with the steps. I was using Ludwig Gorranson’s compositions for the 2020 Nolan film ‘ Tenet’ as a reference, I wanted to create a similar palette. I also applied a tremolo to a field recording of wind. To emphasise the first beat of the bar I used two elements, a reversed clap and a sped up recording of an egg shaker increasing in tempo. Both of these sounds are very high in frequency so they cut through the constant low frequencies from the contact mics. Lastly to add some melody I manipulated a recording of a train engine. This choice was informed by the writings of Pierre Schaeffer on sound retaining meaning (specifically the train and the foundry).

This more musical approach was interesting, I felt I could achieve a more dense level of emotional information but the manipulations detracted from the audiovisual correspondence. The main problem I had with my process was that I had decided to use Protools as a workstation to compose, this resulted in many problems, from instruments not working to audio files not being able to be imported, mainly just a slower and less efficient process of creation.

VISITING PRACTITIONERS

KMRU

KMRU (Joseph Kamaru) is an ambient music artist who has a focus on field recording. He is drawn to meditative and calming music, possibly as a result of his upbringing in the noisy streets of Nairobi. He describes it as a means to ‘slow things down.’

In high school, Joseph moved from the city to a more rural area. This drastic change of sonic environment caused a sensitivity to his surroundings and a focus on his aims. Kamaru started to conduct field recordings, collecting material to use in his music production. He emphasises minimalism as a means of inspiration, the limitation forces you to make the best work with what you have. There is also an emphasis on being open to whatever happens, improvising on synthesisers while not restricting the outcome.

Kamaru is currently doing a three year master’s program in the Berlin university of arts. The transition from Nairobi to Berlin was difficult as the sonic landscapes are so different, Kamaru started to wonder if his sound would change. Joseph’s intention and understanding of field recording developed as he went deeper into the practice. His relationship to the practice was strictly functional, he valued the recording itself, for its sonic qualities and utility for music. Now, Kamaru has started to consider the location as an important aspect of the practice.

‘I feel that sound itself is the best way to learn and understand our surroundings

Kamaru has been making an effort to remove the act of recording occasionally, just to sit and listen as a meditative activity. Sometimes he will bring a notebook and pen to describe his surroundings, this seems to be a method of becoming more perceptually literate. Experience is as important as the act of recording. Occasionally he will draw what he hears as a way to document experience.

Kamaru’s ambient music philosophy is to use elements of recordings in extended ways to create new textures/melodies. The aim of his method is to blend music with field recording to create a sense of insecurity in the listener as to whether music is happening, the field recordings also provide a narrative behind the music. The listener has something to connect to enhancing their experience.

I find KMRU’s work inspiring as its a practice I’m unfamiliar with, I don’t make ambient music and I only rarely conduct field recordings. His idea of giving the listener something more to connect to and giving instruments a narrative is very interesting. I believe this philosophy applies to my interest diegetic underscore, the recordings you compose with carry a narrative that is evident even after manipulation.

ELEMENT 2

Recording session

First thing I did was mark out the sections for the spatial part of the piece. Chair for scale.
Then I installed the recording equipment. It consisted of a 3Dio Binaural mic and two contact mics running as two mono channels.
3Dio Binaural microphone
Contact Microphone
Checking levels

The recording session went well, there was not as many people as I had hoped but the experiment still worked. Eight people attended the experiment, this resulted in a decent ‘crowd’ sound. The first recording went well, the lack of definite tempo and lack of practice made the synchronisation gradual (which is what I was aiming for). I set a tempo of 115bpm for the next recording, this is because I believed it to be a good tempo for composition if I want to layer more musical elements on it. The synchronisation was less gradual with each recording as the participants became more familiar with the action, which I should’ve anticipated but I believe with more participants the synchronisation would take more time. The one thing I forgot to mention to the participants until the last recording was to increase the volume of the footsteps after the synchronisation had occurred, I assumed that this would happen anyway but it didn’t.

Im glad I decided to use contact microphones as I believe those recordings will come in very useful for creative manipulation. My initial idea is to bring the contact mic recording of the synchronisation section up in the LFE, preceding the binaural synchronisation. A subtle foreshadowing of the incoming intensity.

MA COLLABORATION

What remains of Edith Finch

In our first meeting the MA group told us that their game was influenced by this 2017 game ‘What remains of Edith Finch’. This is a first person magical realism game in which the player gradually uncovers the history of the Finch family. The player explores the family home, uncovering clues to generations of the family, the story of their deaths, and the curse that haunts them. The object of this game is not to win, but to spend time immersing yourself in the stories of each Finch’s life.

The player must discover hidden passageways to enter each room, the whole house having an etherial quality, the game has been described as reminiscent of a pop up book. The player is transported from one story to another, each with their own palette. The stories of their deaths are told in very creative and thought provoking ways, including allegories from the perspective of animals, and text from the notes found in the house integrated in the scenery of the story.

Ian Dallas (the game’s head designer) talks about the sound being influenced by films such as Donnie Darko and Under the Skin. He says he listened to the soundtracks as a means to get into a dreaming headspace to conceptualise the game.

I can definitely see how the MA group’s ‘Reverse’ is influenced by this story driven style of game and the etherial quality of Edith Finch. I think this otherworldly palette is a deep well for creative licence and should be very fun to play around with. I am focused primarily on music in my practice, so this will be most of what I do, but I think this otherworldliness is achievable through reverb on the foley and atmospheres. Due to the game’s dream like aesthetic I hope to achieve a level of disorientation or weightlessness in parts of the sound design, our team has an opportunity to contrast this dreamscape with the weighted dreary world of the office. Depending on how the game turns out visually, this could be a very interesting and effective juxtaposition of worlds.