Harry Charlton’s Sound notes

Integrating sounds

Now that we received a video of the game we’re able to start integrating the sounds, this has turned out to be an interesting process. Due to the circumstances of our collaboration everyone in the group has been creating the different elements of the game separately, working from the diagrams and notes provided by the MA group. Although not the best method of collaboration, this has proved to be useful due to the depth of sound required for the game. This way we were able to split the workload and cover a wider spectrum of the game, the draw back to this compromise was a lacking of coherency. There was a general coherency in sound, I believe there is a standardised ‘mini-game sound’ quality. However I feel given more time there could’ve been more coherency. There are three tracks used for the game provided by Pedro, Max and myself. Pedro’s track reflects the style of the mini games, light hearted, synthesiser heavy, playful music. Max contributed a track that provided an appropriate musical backing to the dreamscape of the 2nd level with synthesised game like elements. I feel as though my contribution is the only track that doesn’t have a game aesthetic, I went for a merger between industry and nostalgia, an accurate reflection of the theme of the game, but an aesthetic anomaly. This difficulty was a result of our disjointed collaboration, to remedy this I would have composed with instruments that had a game type timbre and integrated that into the convolution.

Working from the video proposed another level of difficulty as we had recorded and organised our foley as if it would be inputed in FMOD or Unity, by this I mean as single steps and actions. This meant when I came to input these sounds into the video I had to import and arrange individual steps and clicks, a tedious and time consuming process. If I were to undergo this process again from the start with the understanding I have now having seen the full game, I would try to merge game sound with office sound to re-enforce the narrative of the game. This would either mean gradually integrating game sounds into the foley of the office, or integrating office sounds into the dreamscape the connect these two worlds.

We have received the game project file, while I am working on this video to give an overall idea of our sound collaboration, I am going to try and get someone else to attempt to integrate some of our sounds for part of the game using Unity or FMOD. It would be good to integrate our sounds in this way as I believe they would work together better in an immersive and interactive context.

Merging audio using convolution reverb with custom impulse response

Using logic’s space designer I was able to customise an impulse response as a means of merging audio files, as mentioned by Martin Stig Anderson in his lecture.

I wanted to integrate music into the game in an immersive way, so as to emotionally influence the player while maintaining suspension of disbelief. By intwining the melody with something in the game I was able to diegetically anchor the music. The audio I chose to use for this was the sound of a foundry:

I chose this because I believed it shared a contextual/timbral quality with the kaleidoscopic room from the third level, this level being a good summary of the theme and feel of the game. Disorientating, curious and otherworldly.

The first experiment I conducted was applying a convolution reverb to the music, the convolution reverb had the sound of the foundry as it’s impulse response. The result was a strange distortion of the music, into a kind of soundscape. It lost its form and became a kind of fluid representation of the composition the was influenced by the foundry. I would ideally like this influence to be more prominent but I couldn’t find any way to make this the case, even after a lot of tweaking.

I wanted to try the process the opposite way to see if the effect was more successful. I found it hard to understand what each element of modulation in the convolution reverb did, but though general experimentation I managed to gain a basic understanding. The most important thing I didn’t manage to understand was how to control the degree of abstraction away from the original form of the audio. This experimentation had a lot less form than the previous one, the music lost its melodic variation and became one tone. Almost like a summary of the composition layered on the foundry.

I found a compromise by using the ‘music through room convolution’ and layering the original foundry audio with a convolution reverb that had the same impulse response (although the wet and dry levels were adjusted to be less drastic). The idea of this was to have an ‘accurate’ reverb under the original recording, that could blend with the musical reverb. It provides an extra tether to reality.

This experiment was a success, it is a diegetically anchored sound scape composition. I think for this method to be more successful (by that I mean retain more of the diegetic form), I would have to understand the convolution merging technique better and possibly compose the music with the exact same timbre and similar frequencies as the diegetic sound. The composition reflects the game’s themes of nostalgia and self reflection, whilst not taking the player out of the experience, it fits the disorienting world of the game.

Interacting with and listening to game sound

The audiences perception of a sound is dependant on how they listen to it, how one listens can be discussed in terms of Michel Chion’s three modes. The first is casual listening, this is defined as recognising the source of the sound, we instinctively gather information about the world through casual listening. This mode provides the listener with the general characteristics of the objects around them. The second is semantic listening, this is the act of listening and interpreting messages in the sound. This could be seen as not only listening to the words someone speaks, but the tone in their voice. Karen Collins provides the example of the beeping tower in fallout 3, one can casually listen to the beeps and determine that there is a tower in the area and maybe even the distance. One can also listen semantically, providing they understand morse code, and determine a message within the sound. Lastly there is reduced listening, this is focusing in on the auditory qualities of the sound irregardless of the context. As a sound designer we must be conscious that the player can be listening in multiple modes at any time.

Collins goes on to explain the development of these listening modes to accommodate interactive gaming. The musicologist David Huron proposes some additions to Chion’s theory with signal listening, sing along and retentive listening. Signal listening refers to how the player listens in anticipation of a signalling sound. Singalong listening is the act of following along with the sound, an example of this would be karaoke games. Retentive listening is the act of listening with the intention of recalling the information later.

Interactivity with sound in video games is largely dependant not on listening, but the invoking of sound. There is a fluidity to interactivity in game sound, to accommodate for this Collins provides a diagram to separate and identify the different categories of interaction.

Cognitive/psychological interactions are in the centre of the diagram because Collins sees them as a constant phenomenon from which all other interactions take place. This is considered a constant phenomenon by Collins because ‘active reception’ is a form of interactivity (this is informed by Manovich’s reasonings on the misinterpretation of interactivity in computer based media). There is a distinction between two categories of psychological interactions, that which takes place between the player and the game, and that which extends beyond the context of the game itself.

This text was extremely useful as it provides a good basis from which to approach sound design in games. It did stray a lot into more general interactivity in games which I found to be not as focused and valuable but it was still interesting. With a greater understanding of interactivity in games, one can achieve a higher level of it.

Meticulous care was put into the foley for the protagonist of this game, as it is a constant sound throughout the game. There is an option in the game to equip ‘stealth armour’, this makes you less evident to enemies. The care put into the sound design can be seen in this element of the game as the designers did not just turn down the volume of the default armour ‘backpacking’ sounds, rather they rerecorded the foley handling the items more carefully. This is not only a more accurate representation of the sound, it also provides a subtle auditory narrative, caution. It’s this care that enriches the sonic world of the game and therefore the immersion of the experience.

Another example of care is the variation in water sounds, different objects have a specific sounds when dropped into water. The sound is dependant on the size of the object, this is also the case with objects on fire, an arrow on fire sounds like a smaller fire than a larger weapon a blaze. This informs the player of how different fires have different uses in the game. When designing the sonic world, it seems one of the most important thing to consider is how objects interact with different elements in that world. These variations in sounds give the objects a weight in reality.

Currently my practice is more focused on the musical side of sound design, and the music in this game is very well thought out. Each village in the game has a theme that reflects the environment, not only that but each has a day and night version of their theme that transition seamlessly into each other. I think that this is a very smart decision on the sound designers part, it makes the environments informationally denser and enriches the players experience of each town. There is new musical information to be discovered based on the amount of time the player stays in the town. This reminds me of a concept I read about in the article ‘Extending Musical Form Outwards in Space and Time: Compositional strategies in sound art and audiovisual installations‘. In the same way installations should be an invitation for perceptual investigation, so should games. As a sound designer for games we must consider the modulating factor of the agency of the mobile visitor, it is a modulating factor for our creations.

I think one of the most effective use of sound design in this game is the interweaving of music and action in combat situations. The game is designed to play combat themes when the player gets within a certain distance of an enemy, there are many variations of this theme depending on the difficulty of the opponent. The most interesting part of this design, in my opinion, is the fact that musical accents are added to the theme when a player lands a hit on an opponent. These accents are also modulated by the strength of the hits, the stronger the hit, the louder the musical accent. These accents are added to the beat of the bar directly after the hit. This piece of innovation is genius in my opinion, as it blurs the line between musical score and action. The musical score is there as a way to influence the player, to amp them up in combat. However the player is influencing the score while engaging in the combat, resulting in a kind of feedback loop of exhilaration.

There are multiple sound design techniques implemented in this game that I wish to engage in, and researching them has given me a better appreciation of how sound design can provide a lot of information to the player.

What remains of Edith Finch

In our first meeting the MA group told us that their game was influenced by this 2017 game ‘What remains of Edith Finch’. This is a first person magical realism game in which the player gradually uncovers the history of the Finch family. The player explores the family home, uncovering clues to generations of the family, the story of their deaths, and the curse that haunts them. The object of this game is not to win, but to spend time immersing yourself in the stories of each Finch’s life.

The player must discover hidden passageways to enter each room, the whole house having an etherial quality, the game has been described as reminiscent of a pop up book. The player is transported from one story to another, each with their own palette. The stories of their deaths are told in very creative and thought provoking ways, including allegories from the perspective of animals, and text from the notes found in the house integrated in the scenery of the story.

Ian Dallas (the game’s head designer) talks about the sound being influenced by films such as Donnie Darko and Under the Skin. He says he listened to the soundtracks as a means to get into a dreaming headspace to conceptualise the game.

I can definitely see how the MA group’s ‘Reverse’ is influenced by this story driven style of game and the etherial quality of Edith Finch. I think this otherworldly palette is a deep well for creative licence and should be very fun to play around with. I am focused primarily on music in my practice, so this will be most of what I do, but I think this otherworldliness is achievable through reverb on the foley and atmospheres. Due to the game’s dream like aesthetic I hope to achieve a level of disorientation or weightlessness in parts of the sound design, our team has an opportunity to contrast this dreamscape with the weighted dreary world of the office. Depending on how the game turns out visually, this could be a very interesting and effective juxtaposition of worlds.