Category: Y2 COLLABORATION

Integrating sounds

Now that we received a video of the game we’re able to start integrating the sounds, this has turned out to be an interesting process. Due to the circumstances of our collaboration everyone in the group has been creating the different elements of the game separately, working from the diagrams and notes provided by the MA group. Although not the best method of collaboration, this has proved to be useful due to the depth of sound required for the game. This way we were able to split the workload and cover a wider spectrum of the game, the draw back to this compromise was a lacking of coherency. There was a general coherency in sound, I believe there is a standardised ‘mini-game sound’ quality. However I feel given more time there could’ve been more coherency. There are three tracks used for the game provided by Pedro, Max and myself. Pedro’s track reflects the style of the mini games, light hearted, synthesiser heavy, playful music. Max contributed a track that provided an appropriate musical backing to the dreamscape of the 2nd level with synthesised game like elements. I feel as though my contribution is the only track that doesn’t have a game aesthetic, I went for a merger between industry and nostalgia, an accurate reflection of the theme of the game, but an aesthetic anomaly. This difficulty was a result of our disjointed collaboration, to remedy this I would have composed with instruments that had a game type timbre and integrated that into the convolution.

Working from the video proposed another level of difficulty as we had recorded and organised our foley as if it would be inputed in FMOD or Unity, by this I mean as single steps and actions. This meant when I came to input these sounds into the video I had to import and arrange individual steps and clicks, a tedious and time consuming process. If I were to undergo this process again from the start with the understanding I have now having seen the full game, I would try to merge game sound with office sound to re-enforce the narrative of the game. This would either mean gradually integrating game sounds into the foley of the office, or integrating office sounds into the dreamscape the connect these two worlds.

We have received the game project file, while I am working on this video to give an overall idea of our sound collaboration, I am going to try and get someone else to attempt to integrate some of our sounds for part of the game using Unity or FMOD. It would be good to integrate our sounds in this way as I believe they would work together better in an immersive and interactive context.

Merging audio using convolution reverb with custom impulse response

Using logic’s space designer I was able to customise an impulse response as a means of merging audio files, as mentioned by Martin Stig Anderson in his lecture.

I wanted to integrate music into the game in an immersive way, so as to emotionally influence the player while maintaining suspension of disbelief. By intwining the melody with something in the game I was able to diegetically anchor the music. The audio I chose to use for this was the sound of a foundry:

I chose this because I believed it shared a contextual/timbral quality with the kaleidoscopic room from the third level, this level being a good summary of the theme and feel of the game. Disorientating, curious and otherworldly.

The first experiment I conducted was applying a convolution reverb to the music, the convolution reverb had the sound of the foundry as it’s impulse response. The result was a strange distortion of the music, into a kind of soundscape. It lost its form and became a kind of fluid representation of the composition the was influenced by the foundry. I would ideally like this influence to be more prominent but I couldn’t find any way to make this the case, even after a lot of tweaking.

I wanted to try the process the opposite way to see if the effect was more successful. I found it hard to understand what each element of modulation in the convolution reverb did, but though general experimentation I managed to gain a basic understanding. The most important thing I didn’t manage to understand was how to control the degree of abstraction away from the original form of the audio. This experimentation had a lot less form than the previous one, the music lost its melodic variation and became one tone. Almost like a summary of the composition layered on the foundry.

I found a compromise by using the ‘music through room convolution’ and layering the original foundry audio with a convolution reverb that had the same impulse response (although the wet and dry levels were adjusted to be less drastic). The idea of this was to have an ‘accurate’ reverb under the original recording, that could blend with the musical reverb. It provides an extra tether to reality.

This experiment was a success, it is a diegetically anchored sound scape composition. I think for this method to be more successful (by that I mean retain more of the diegetic form), I would have to understand the convolution merging technique better and possibly compose the music with the exact same timbre and similar frequencies as the diegetic sound. The composition reflects the game’s themes of nostalgia and self reflection, whilst not taking the player out of the experience, it fits the disorienting world of the game.

Interacting with and listening to game sound

The audiences perception of a sound is dependant on how they listen to it, how one listens can be discussed in terms of Michel Chion’s three modes. The first is casual listening, this is defined as recognising the source of the sound, we instinctively gather information about the world through casual listening. This mode provides the listener with the general characteristics of the objects around them. The second is semantic listening, this is the act of listening and interpreting messages in the sound. This could be seen as not only listening to the words someone speaks, but the tone in their voice. Karen Collins provides the example of the beeping tower in fallout 3, one can casually listen to the beeps and determine that there is a tower in the area and maybe even the distance. One can also listen semantically, providing they understand morse code, and determine a message within the sound. Lastly there is reduced listening, this is focusing in on the auditory qualities of the sound irregardless of the context. As a sound designer we must be conscious that the player can be listening in multiple modes at any time.

Collins goes on to explain the development of these listening modes to accommodate interactive gaming. The musicologist David Huron proposes some additions to Chion’s theory with signal listening, sing along and retentive listening. Signal listening refers to how the player listens in anticipation of a signalling sound. Singalong listening is the act of following along with the sound, an example of this would be karaoke games. Retentive listening is the act of listening with the intention of recalling the information later.

Interactivity with sound in video games is largely dependant not on listening, but the invoking of sound. There is a fluidity to interactivity in game sound, to accommodate for this Collins provides a diagram to separate and identify the different categories of interaction.

Cognitive/psychological interactions are in the centre of the diagram because Collins sees them as a constant phenomenon from which all other interactions take place. This is considered a constant phenomenon by Collins because ‘active reception’ is a form of interactivity (this is informed by Manovich’s reasonings on the misinterpretation of interactivity in computer based media). There is a distinction between two categories of psychological interactions, that which takes place between the player and the game, and that which extends beyond the context of the game itself.

This text was extremely useful as it provides a good basis from which to approach sound design in games. It did stray a lot into more general interactivity in games which I found to be not as focused and valuable but it was still interesting. With a greater understanding of interactivity in games, one can achieve a higher level of it.

Meticulous care was put into the foley for the protagonist of this game, as it is a constant sound throughout the game. There is an option in the game to equip ‘stealth armour’, this makes you less evident to enemies. The care put into the sound design can be seen in this element of the game as the designers did not just turn down the volume of the default armour ‘backpacking’ sounds, rather they rerecorded the foley handling the items more carefully. This is not only a more accurate representation of the sound, it also provides a subtle auditory narrative, caution. It’s this care that enriches the sonic world of the game and therefore the immersion of the experience.

Another example of care is the variation in water sounds, different objects have a specific sounds when dropped into water. The sound is dependant on the size of the object, this is also the case with objects on fire, an arrow on fire sounds like a smaller fire than a larger weapon a blaze. This informs the player of how different fires have different uses in the game. When designing the sonic world, it seems one of the most important thing to consider is how objects interact with different elements in that world. These variations in sounds give the objects a weight in reality.

Currently my practice is more focused on the musical side of sound design, and the music in this game is very well thought out. Each village in the game has a theme that reflects the environment, not only that but each has a day and night version of their theme that transition seamlessly into each other. I think that this is a very smart decision on the sound designers part, it makes the environments informationally denser and enriches the players experience of each town. There is new musical information to be discovered based on the amount of time the player stays in the town. This reminds me of a concept I read about in the article ‘Extending Musical Form Outwards in Space and Time: Compositional strategies in sound art and audiovisual installations‘. In the same way installations should be an invitation for perceptual investigation, so should games. As a sound designer for games we must consider the modulating factor of the agency of the mobile visitor, it is a modulating factor for our creations.

I think one of the most effective use of sound design in this game is the interweaving of music and action in combat situations. The game is designed to play combat themes when the player gets within a certain distance of an enemy, there are many variations of this theme depending on the difficulty of the opponent. The most interesting part of this design, in my opinion, is the fact that musical accents are added to the theme when a player lands a hit on an opponent. These accents are also modulated by the strength of the hits, the stronger the hit, the louder the musical accent. These accents are added to the beat of the bar directly after the hit. This piece of innovation is genius in my opinion, as it blurs the line between musical score and action. The musical score is there as a way to influence the player, to amp them up in combat. However the player is influencing the score while engaging in the combat, resulting in a kind of feedback loop of exhilaration.

There are multiple sound design techniques implemented in this game that I wish to engage in, and researching them has given me a better appreciation of how sound design can provide a lot of information to the player.

What remains of Edith Finch

In our first meeting the MA group told us that their game was influenced by this 2017 game ‘What remains of Edith Finch’. This is a first person magical realism game in which the player gradually uncovers the history of the Finch family. The player explores the family home, uncovering clues to generations of the family, the story of their deaths, and the curse that haunts them. The object of this game is not to win, but to spend time immersing yourself in the stories of each Finch’s life.

The player must discover hidden passageways to enter each room, the whole house having an etherial quality, the game has been described as reminiscent of a pop up book. The player is transported from one story to another, each with their own palette. The stories of their deaths are told in very creative and thought provoking ways, including allegories from the perspective of animals, and text from the notes found in the house integrated in the scenery of the story.

Ian Dallas (the game’s head designer) talks about the sound being influenced by films such as Donnie Darko and Under the Skin. He says he listened to the soundtracks as a means to get into a dreaming headspace to conceptualise the game.

I can definitely see how the MA group’s ‘Reverse’ is influenced by this story driven style of game and the etherial quality of Edith Finch. I think this otherworldly palette is a deep well for creative licence and should be very fun to play around with. I am focused primarily on music in my practice, so this will be most of what I do, but I think this otherworldliness is achievable through reverb on the foley and atmospheres. Due to the game’s dream like aesthetic I hope to achieve a level of disorientation or weightlessness in parts of the sound design, our team has an opportunity to contrast this dreamscape with the weighted dreary world of the office. Depending on how the game turns out visually, this could be a very interesting and effective juxtaposition of worlds.

Video game sound designers have a unique task in the world of sound design. In a film, the sound designer knows exactly what will happen, but in a game the designer must account for the player’s agency. The world will change depending on the player’s interaction with it. This is ‘adaptive audio’, audio that adapts to player input. An example of adaptive audio in game design is footsteps, they must first be tied to the event of walking (then the player inputs the signal to walk), the audio must then be matched to the walking animation. The necessary work to achieve audiovisual correspondence in games is far more than film. There is another degree of adaptivity beyond this, adapting to environment. The footsteps must change sound depending on where the player moves their character. This is especially important when there is an informative element to it, for instance if ice is slippery there should be an auditory cue for the player.

Another degree of adaptivity is how the sounds we implement interact with the environment in the game. Inaccurate reverb will bring the player out of the game, ruining the immersion. As sound designers our job is to immerse the player, and to avoid distracting their attention from where it ought to be.

To solve these problems we must include variation within the sounds triggered by the event. This can be done though designating certain reverbs to certain zones within the game. We can also implement gradual gradual transitions that is dependant on movement thought environments, this is usually tied to music transitions from the different locations. The most effective utilisation of adaptive audio, in my opinion, is when designers pitch sound effects to key of the background music. This results in a merger between diegetic interactive sound and the non-diegetic sonic/melodic world of the game. I hope to have the opportunity to explore implementing these parameters in FMOD.

Five nights at Freddy’s is a horror game in which the player is trapped in the security office of a pizzeria while animatronic creatures attempt to kill them. The horror feeling is built out of two concepts, absence of action and absence of information.

The absence of information refers to the fact that the animatronic creatures move when the player is not looking at them, and so the player must rely on clues as to their whereabouts. The player has access to some obvious visual clues such as the CCTV system, but a lot of the information is subtilely delivered to the player through the audio of the game. The audio is a more complicated source of information as some audio helps you and some deceives you, this adds to unsettling nightmare like sense of not knowing.

Within the game sound there is advantageous audio, this is the audio that helps the player with their goal of survival. These sounds do not reveal themselves straight away, when the player begins the game the sounds present themselves as chaotic. As more time is spent in the game, the player gains an understanding of the auditory patterns and their listening transitions from casual to semantic, deducing clues from the patterns. An example of this is the sound of pots and pans, at first this sound seems like an element of ambience, put in the game to induce panic in the player. However as the player interacts with the game more they will find that this sound becomes louder when they look at the kitchen camera (which has broken visuals, its audio only). When coupled with the player noticing that one one of the animatronics is missing, they can deduce that the animatronic is in the kitchen. These unsettling sounds have a purpose in the gameplay.

The most interesting part of the sound design for this game is the detrimental audio, the audio that is there to be a distraction from the truth. One element of this is the choice to have very poor transmission of audio from the cameras, the audio is filtered and manipulated heavily as a means to muddy the information. There are also sounds that are implemented simply to get in the way, to demand attention at a time of intense audio sensitivity. This is the mechanical sound of the cameras turning or the intermittent static. All of this sound is also lightly obscured by the fan in the office, this is a constant sound that is off a similar frequency to the audio from the cameras. These sound design choices are a very clever means to disorient and frustrate the player in a similar way to how one might experience a nightmare, like the feeling of running and going nowhere, there are simple little things that make success so much harder. When paired with a life or death scenario the feeling is intense frustration.

I think that sound has a unique capability in horror games specifically. In my opinion audio hallucination is much less obvious to the subject than visual hallucination, yet holds a great deal of weight in our sense of orientation in reality. We understand a lot of our environment through listening, and in a game context where this audio can not only be manipulated but also interacted with (giving it a greater weight in reality), sound designers can achieve a uniquely disorientating experience.

Mechanical Bird Sound First Draft

The first thing I did when constructing this sound was identify the different samples I could use. The sound was divided into three elements, natural, mechanical cog and Motor. I started by finding audio of a bird flapping its wings, this is what I would use as a blueprint to lay the mechanical sounds on top of. I tried to match the real sound source’s size to the fictional object’s size. The first mechanical sound I added was the sound of the cog pulling the wing down, for this I used layered camera shutters, because its a high velocity small mechanism. The second element of the mechanical bird was a motor, I imagined that a motor would be necessary for a mechanical system. The only motor that would fit in a mechanical would be a watch motor. These layered created a textured mechanical system to place under the natural sounds. In the interest of time I created a 6 flap loop, when I revisit this sound I will find more diverse samples and create a longer loop to make the sound more believable.

The natural sound provided the hardest problem when designing this sound. I found that recordings of real birds had a Doppler effect as the birds were always flying past the microphone, this was unusable for me as I needed a steady flapping. The only alternative was foley recordings I found in sound libraries, the drawback to this was they all had a very artificial quality to them. Another problem I experienced was determining the amount of low frequency that should be present in the flapping. From the perspective of a person the air displacement wouldn’t be that much, therefore there wouldn’t be a lot of low frequency in the sound. However, from the perspective of the bird there would be much more low frequency. The problem I face is that the amount of low frequency in the wing flap is an auditory signifier of the size of the bird, how do you express the size of the bird accurately?

Martin Stig Anderson Lecture

Martin Stig Anderson begins his lecture by reading a couple of reviews people had written about his work on the 2010 game ‘Limbo’. As you can see the reviews are completely polarised as to whether there is music in the game. Music can be much more than rhythm, harmony and melody. Ambient and environmental noises are the feature of the show. 

Anderson lays the foundation of his practice with the definition of music as ‘organised sound’, then going on to talk about Pierre Schaeffer’s music concrete. There must, however, be a distinction between soundscape composition and music concrete, although they are essentially the same practice there is an important varying factor. In the practice of soundscape composition, ‘the original sounds must stay recognisable and the listeners contextual and symbolic associations should be invoked’. – Barry Truax

Soundscape composition exists on a spectrum with music concrete, between contextual immersion, abstracted middle ground and absolute abstraction. Music concrete is the movement from a abstracted middle ground to absolute abstraction, and soundscape composition a movement from contextual immersion to abstracted middle ground. 

A technique Anderson uses to merge orchestral elements into the diegesis of the film is spectral interpolation, for the game limbo he did this with an orchestral recording and a recording of a foundry. ‘It retains the nuances of the orchestral but its not really there at the same time’. This was for one of many rotating rooms in the game. The second was a room of circular saws, for this he used a recording of a bowed cymbal, yet again he interpolated the orchestral sound with this recording. This results in a diegetic sound with complete contextual immersion, which contains harmonic/melodic qualities. Anderson uses source filtering and convolution techniques to merge the two audio