Curating Accidents and Mistakes

Curating Accidents and Mistakes

As sound designers and mixers we struggle to get some level of control; control of our working situations and our tools. The funny thing is, sometimes not precisely controlling the tools we work with can lead to the most interesting results. Often an imprecise tool is exactly what we need. Like this one.

Tools of the Trade?

I’ve been doing more and more landscape painting over the last decade or so when I’m not working on movies. I’m not a great painter, but I’ve learned lots of things about it by diving in and giving myself permission to make lots of mistakes, then doing my best to figure out what each mistake can teach me.

What we can learn from impressionist artists...

This is part of a painting by Paul Cezanne. If you want to appreciate the beauty of a painting it’s rarely a good idea to look at it extremely closely, but that’s what I suggest you do with this one. It’s basically all “mistakes” and “accidents,” but Cezanne curated each one very, very carefully. If you zoom in far enough so that any of these individual apples fills the screen, you wouldn’t even guess you were looking at an apple. It’s just a seemingly random bunch of shapes and colors. But the overall result when you look at the whole thing from a distance is a stylized, beautiful, and compelling Impressionist masterpiece.

So, what’s the lesson here for sound?

I believe it’s best to think of sound design as more of an impressionist art form than one focused on realism. The goal should not be to try to reproduce reality. It should be to offer a twist on reality that expresses a set of emotions helpful to the overall storytelling.

Employing the most precise tools and using them in the most controlled way in every moment isn’t the best way to find those useful twists. Learning how to curate your mistakes and accidents is a better approach.

When I’m working on the sound for a shot of trees blowing in the wind, I never start by adding a sound for each limb moving, one at a time. I find a recording of wind-blown trees, and arbitrarily lay that up to the picture. There will almost always be at least one or two moments where something in the sound speaks to the visuals in a way I could not have matched by cutting each limb separately. Conversely, there will always be visual elements that aren’t covered by the general tree sound. I then try a similar approach, putting another, slightly more focused sound in rough sync with the moment I need to cover. Very often another piece of magic happens, and a sound I wouldn’t have thought to look for will express exactly what I need to express about that moment.

I’ve used this approach with the waves in Cast Away, the bicycle disintegrating in Forrest Gump, Elastigirl stretching in The Incredibles, and the bathroom plumbing going crazy in the Super Mario Bros. Movie.

​A rough brush, applied quickly, then carefully edited. It works!

Avoiding Spectral Logjams

Avoiding Spectral Logjams

Having designed, edited, and mixed the sound for hundreds of action sequences, I can testify that the biggest problem to overcome is usually having too much sound.

The “too much sound” comes in several forms. The most basic one is simply having too many simultaneous, individual sounds competing for attention. We sound designers tend to be afraid of having produced too few sounds for a given sequence, and in trying to allay that fear, we produce too many.

The classic mistake is to throw several mediocre sounds at a given moment in the hope that when they play together some kind of magic will happen to turn the moment into something better than mediocre. It rarely does, and you wind up with a bunch of average sounds masking each other into noise. The solution is to have fewer sounds and better sounds. That’s where creativity is necessary, as opposed to throwing a ton of gunk at the canvas and praying the mixer will somehow make it beautiful. Yep. I’ve been guilty too.

"Too much sound" is something more specific than too many sounds...

But another form of “too much sound” is something more specific than too many sounds, it’s too much sound in a given spectrum. In my experience, the part of the audio spectrum between about 200Hz and 800Hz is usually not the problem. The two problem areas tend to be below 200Hz and above 1K, especially above 3K.

For some reason we often conclude that low frequencies and high frequencies are where most of the drama is in sound effects, that those parts of the spectrum are the ones that get people’s attention. So, we pile it on in those areas in action sequences: constant booming bass that feels like low-passed pink noise and constant highs that basically feel like unrelenting, uncorrelated white noise. It’s not dramatic, dynamic, or even coherent. It’s just damn loud.

My recommendation for how to avoid this common, nasty situation that doesn’t sound good and will usually get you into trouble with Directors (or whoever is creatively in charge) is to not only reduce the number of simultaneous sounds, and to come up with individual sounds that have more natural intrinsic drama, but also to listen carefully for unnecessary sonic competition in particular frequency ranges.

Certain sounds will have something stellar going on in the highest frequency bands. Showcase that sound by filtering or eq’ing down the high frequencies in any other simultaneous sound that could mask the star of the moment. Same for the low end. The subwoofer can shine, but not if you neutralize it by sending three, six, eight sounds to it simultaneously.

​Lots of this happens in the mix, of course, but my advice is to save the mixer some time by preparing sounds so that when all the faders are initially put up; the mixer isn’t confronted with a wall of blasting meaninglessness.

Good Luck!

Sound Design and AI

Sound Design and Artificial Intelligence

As I write this in late April of 2024, artificial intelligence has not had a significant impact on sound effects design for film, video, or games. But I specify the date because I expect the situation to change in the not-too-distant future. A.I. is already being used in various kinds of voice and music work, with varying degrees of success.

The most promising use of a.i. for sound effects that I know of has been done by a company called ElevenLabs. But their process is not available yet. They say they have about 11,000 people on their waiting list.

ElevenLabs have posted a demonstration in which they claim to have generated sound effects for a series of videos produced by Open A.I.’s “Sora” program, which creates moving images from a text input. The quality of the sound design in the ElevenLabs Sora demo is mediocre at best in my opinion, but the fact that they can do it at all is impressive.

Legal and Ethical questions...

As with all forms of A.I., there are lots of legal and ethical questions that need to be addressed. One of the foremost has to do with compensating the individual people and companies who produced the material used to train the a.i. Another major issue is the number of human jobs that will be eliminated if and when, most likely when, A.I. does begin to make a serious impact on sound design work.

To be honest, if I could make A.I. go away, I would. But it won’t go away. It will inevitably get more and more sophisticated, eliminating more human jobs in the process. My advice to people who have a career or want a career in sound design is to learn everything you can about A.I. for sound. That will put you in the best position to use it, instead of being replaced by it.

The jobs on projects at the low end of the budget scale will be the first lost to A.I. That’s a shame, because it’s those projects where people new to the industry usually get their foot in the door. The high-end, high-profile projects will be the last to adopt A.I. sound design because they can afford the added value they get from human creative involvement.

The creative level of the sound work in the existing Sora videos is what I would call “see a dog, hear a dog.” Obvious and not very interesting interpretations. You can outdo the current and near-term future level of A.I. sound design by going beyond the obvious.

If you are seeing a truck traveling down a road, don’t settle for finding or making an audio recording of that kind of truck. Make the truck a character by using non-obvious elements in addition to the obvious ones, and by finding ways to add believable dynamics to the sound that will contribute to the story by tweaking the imagination of the audience.

That level of sound design creativity won’t be in A.I.’s repertoire anytime soon.
Sound Ideas and A.I

Making It Real

Making It Real and Making It Feel

When I was working on the Bob Zemeckis film Cast Away it was very difficult coming up with sounds for the waves on the beach when the Tom Hanks character is alone on the island. The sound recorded while they were shooting those beach scenes in Fiji was almost all useless. It was just the boring “noise” of constant wave breaking in all those shots. It sounded like a giant pink noise generator.

We had to invent a sound for every single wave...

So, we had to invent the sound for every single wave we see in the movie, as well as the hundreds that we don’t see. The waves needed to be a character in the movie, an interesting and totally believable character. It turned out that both the interesting aspect and the believable aspect were harder to achieve than I thought they would be. The sound of the waves needed to mirror the Hanks character’s emotions. When he was calm, they needed to be calm, and when they were intense, he was intense. Each wave also needed to present its own little story with a beginning, middle, and end.

I started the wave work by gathering all kinds of water sounds, not just waves. That’s because I knew from the beginning that I would need to build each wave with elements, not just go out and record real ocean waves. Though I did that too. At least a month was spent going up and down the West Coast of the USA collecting wave sounds, from huge ones to tiny ones. The holy grail was always waves that were hitting the beach at ninety degrees, pretty close to shore. In that situation you get a nice pause between each wave break during which you can hear the fizz of all the bubbles settling into the sand. I made lots of recordings where I focused mainly on getting that fizzing. Though I also faked some fizz elements with things like Alka Seltzer tablets in a studio.

I wanted to make each wave dynamic, changing over time in volume and timbre. Each needed to sound unique, but still in the same family as its neighbors. That’s the “feel” part of the character I was looking for.

But the sound for each wave we see in the movie, and we see lots of them, needed to feel real too. You’d think that waves would be easy to edit. Nope. That’s because each wave is different: different amounts of bubbles, different amounts of impact, different amounts of curl, etc. That’s why we had to design the sound for each wave individually in order to make it believable and real.

​An average wave we see in the movie was made of at least three or four elements. Everything from actual ocean beach waves to water being poured into a swimming pool (to get the watery “curl and flow” sound), to Alka Seltzer, to cannons firing and explosions for the big wave impacts, and beyond. Myself and the sound effects editors would pass wave sequences back and forth, and each person would contribute something new in each iteration that would help sell the sound. It was difficult, fun, and wound up getting us an Oscar nomination that year the following spring. It’s still hard for me to look at a real wave and avoid thinking about how I could improve its sound.

Storytelling Functions of Sound

When someone asks “What does sound do in a movie” most of us are at least a little confused about how to answer that question. Obviously both music and sound effects can make you feel a certain way. Sound effects can clearly add realism to a scene; make it more exciting, more mysterious, or funny. Same for music. But beyond those obvious jobs, it can be hard to specifically say what sound is doing.

Suggesting a Mood or Defining a Place

Here are some of sounds jobs. Very often a given sound will be doing two or more of these at the same time:
    - suggest a mood, evoke a feeling
    - set or modify a pace
    - indicate a geographical locale
    - indicate a historical period
    - indicate changes in time
    - connect otherwise unconnected ideas, characters, places, images, or moments
    - define a character
    - clarify the plot
    - heighten realism or diminish it
    - heighten ambiguity or diminish it
    - draw attention to a detail, or away from it
    - smooth otherwise abrupt changes between shots or scenes
    - emphasize a transition for dramatic effect
    - describe an acoustic space
    - startle or soothe
    - exaggerate action or mediate it
    - indicate the geography within a scene*

                          *(for example: most jungle shots look the same, but sound can tell us
                             who is where relative to the camera when we cut from one jungle
                             location to another two hundred feet away)

If two different sounds are competing while trying to accomplish any one of these jobs, that can be a red flag, because unless they are orchestrated well together, there’s a good chance that each is undercutting the impact of the other rather than each amplifying the other.

This often happens when music and sound effects are attempting to make essentially the same statement at the same time. One of the most difficult aspects of sound mixing is finding ways to feature one sound or set of sounds in each moment and shifting that focus so that the audience feels they are hearing everything they are supposed to be hearing.

​Simply playing every sound loud is usually a disaster unless the goal is to create a wall of noise, which sometimes it is.