In my opinion, while VR is awesome on a visual level, what has yet to truly be realized is the audio potential. I think we have only scratched the surface, and while video and digital imaging is still a relatively new technology, audio, on the other hand, has had a long time to evolve.
Visually, VR still has not reached the level where your eyes stop distinguishing real from simulated. There is simply not enough pixel density and performance for that yet. (Some experts think we need to have at least 16k per eye before that happens!) So, basically, the visual uncanny valley will still be around for a while.
Unless you are a super audiophile, however, digital audio has progressed to the point where a good engineer can make a recording that is very hard to distinguish from reality.
Spatial audio, when mixed with simple VR can add a totally new level of realism to an experience, and I am starting to see some exciting new examples of this appear.
An audio engineer named Aaron Streight recently reached out to me and shared an awesome new 360° video he did of recording artist Ken Yates, which was posted on Facebook.
Despite being a beautiful song, this is a great example of how a simple 360° video can be used to provide context to the audio sources, and a combination of some well placed mics can really enhance the listening experience, simply by turning your head.
Here is a direct link to the video
Aaron’s technique was not only to capture the visual performance, but he recorded in a way that used spherical audio so that when you focus on specific musicians or instruments, the audio changes focus. Tne effect is very subtle, but I encourage you to put on a VR headset, put on your headphones and sit back and listen. A real treat.
Here is a bit of detail from Aaron about how he recorded the piece:
There were many considerations that went into creating this 360 recording for Ken Yates. This performance was recorded with a specific focus on VR audio and mixing sound in 360. The aim was to give the spatial experience of sitting in the middle of a folk performance, by having the performers positioned around the mic and camera.
In order for a good mix of both spherical audio, and localization of individual sound sources, two recording streams had to be used. The first source was the Tetramic, an ambisonic microphone made by Core Sound. This mic accounted for roughly 75% of the final mix and acted as a substantial “room mic” feed on which to build the rest of the recording on top.
The Tetramic captures sound in a unique manner called A-Format, a speaker-independent representation of the sound field, which must be then converted to B-Format before the mixing stage in order to properly send spherical sound to your headphones. The Tetramic was mounted on the camera, and for all intents and purposes, it was treated as the “ears” of the recording. However, due to the volume inequality of the natural mix from the Tetramic – individual mics were also needed to boost the performer’s voices and instruments.
For this, each musician and instrument were miced up conventionally and mixed into the recording like a regular recording session. However, within the Facebook 360 Audio Workstation, the treatment of these recording channels is much different. Each individual sound source in the mix needs localization, as well as spatialization, relative to what’s being heard. In the mix, these spatial coordinates are placed using the azimuth, distance, elevation, and Doppler effect controls to properly articulate where the sound is coming from. Once the 360 mix was sounding good, there was one last quality that was important to test – and that was the Focus.
Focus is the level to which the non-viewable sound field is dimmed in volume, in order to give the sensation of focusing on what you’re viewing and hearing in front of you. Once the focus is set, the master audio is then synced or “muxed” with the VR video and is ready to experience. These are just some of the recording considerations needed when recording music live in 360.
—————–
I’ll have to admit that I am an audio production guy, so I love this kind of stuff. If anyone has any VR related or spatial audio related topics they think I might find interesting, please share.
Sean Earley is the Executive Editor of AR/VR Magazine & co-founder of RobotSpaceship Podcast Network. He is the Director of New Biz Development and Publishing at KEMWEB, a musician, producer & consultant. He loves guitars, VR and coffee.
