Around different corners of the world (virtual or not), audio is the top priority in driving home a truly immersive experience. It adds depth and realism and delivers an emotional weight to the story. This could be by providing auditory cues that complement the visuals to how sound travels in the space. In the last few years, there has been a specific mixing of audio that replicates the way we perceive sound in the real world: spatial audio. Spatial audio simulates the direction, distance, and movement of sound sources to enhance the sense of presence, causing the audio environment more lifelike. In mixed reality applications, the accuracy of positional audio cues is crucial for situational awareness and equally for gameplay. For example, let’s say you want to build a trade show in the metaverse. An essential part of making the whole experience feel real is the people! By implementing spatial audio into your experience, you can hear the audio coming from all around you, gaining a sense of space and allowing sounds to appear closer or farther away. This depth will make your users feel the energy that is needed for building these types of scenarios.
As Dolby, you know for providing this type of experience at your cinema, home theater, or music streaming services; however, with game engine plugins, we are giving you the power to integrate spatial chat into your experience! Spatial chat is communicating in a virtual world incorporating spatial audio technology. It allows participants to communicate with each other spatially accurately, where audio reflects the position and distance of each speaker. This has numerous benefits; enhanced collaboration and privacy/ proximity are within the mix. Now, with all this talk, let’s create an experience ourselves!
In this Unity project, you would be able to set up local multiplayer and hear the voices be spatialized with our communications plugin. To follow along with the tutorial, make sure to download this repository, as it will be what is used in this lesson.
Requirements
- Apple MacOS x64/arm64 or Microsoft Windows 10+ x64.
- A Dolby.io account. If you do not have an account, you can sign up for a free account.
- The client access token copied from the Dolby.io dashboard. To create the token, log into the Dolby.io dashboard, create an application, and navigate to the API keys section.
- Install git-lfs command line tool on your computer if you wish to install directly using the GitHub link.
Downloading the SDK
With the project downloaded, navigate to Unity Hub, select Open, and browse to the project’s directory on your local machine. Ensure the project is unzipped before opening it on the game engine. The editor version used for this project is 2021.3.10f1; our SDK works with any Unity version as long as Visual Scripting is installed. Bolt Visual Scripting will continue to support projects on Unity 2018, 2019, and 2020 LTS versions. A good way to check if it is integrated into your project, go to Window > Package Manager and search Visual Scripting in the packages from your project. Otherwise, at the top of the Package Manager window, there will be a “+” button; you will select “Add package by name…” and enter “com.unity.visualscripting”. This is also where we will add Dolby.io‘s spatial chat plugin. In the same “+” button, select instead “Add package from git URL…” and paste this: https://github.com/DolbyIO/comms-sdk-unity.git.
After it finishes loading, it will show up as DolbyIO Communications version 1.1.1. This is the same location where you could update the plugin in the future as you follow along with future guides from this SDK.
Routing The Spatial Voices
Inside our Demo World playground, we are going to add an empty GameObject and name it “Multiplayers”. Here we will add two components, Dolby.io Manager and Script Machine. The Dolby.io Manager is a script that creates all of the magic of spatial for the conversations that will be going on. For example, we could modify the position and direction duration as it is the elapsed time between the two users to set the position or the direction. For the Audio Listener, let’s select the MainCamera, which will allow the use of the computer’s mic. Within the Script Machine component, we will create a new graph named “Multiplayer”. Edit the Graph to enter the script!
First thing is first, click on On Start, and on the left side inside the Graph Inspector, check off the Coroutine box. This will allow for an iteration instead of completing the task immediately. Next, right-click on the open canvas and, type Initialize, attach it to the On Start. Right-click again and select Conference and link this to the Initialize. Next, make sure the Spatial Audio Style is Shared. There are two scenarios: Shared and Individual. With Shared, all participants contribute to the shared scene so they are heard from the positions they set. In the case of Individual, each client explicitly configures the spatial audio position of every remote participant for themselves. In the Conference Alias, you could input any preferred name (“Multiplayer Chat”). Lastly, we need to input the Dolby.io credentials!
Getting a Dolby.io Token App
Let’s put a pause before adding the Access Token graph to Initialize, and navigate over to Dolby.io. If you are a past user, log in, but if this is your first use of the APIs, then welcome! We offer 50mins for free when you sign up, which should be more than enough to continue with this project and do more experimentation on your own. Inside the dashboard, travel over to the Communications & Media tab at the top. Press on + CREATE APP and name it “unity-spatial-app”.
Click inside the application to see the App key and secret and the Client access token. Either one of these authentications will work with powering the plugin. For a secure authentication, select the Token since if the project is deployed for an audience, then your account information would be exposed with the App key and secret. However, if this is not an issue and is only for your learning, then the App key and secret would work fine.
For the sake of this project, I will select the Token to avoid the exposure of my credentials. Returning to the Script Graph in Unity, right-click and select the String Literal node and paste the Client access token. This Token will expire in 24 hrs, which means that I need to refresh it after each day of use. Next, attach the String node to the Initialize to the Access Token connection. For those that select the other option, add the Get Token node, paste the key and secret, and attach it to the Access Token instead. We are done!
Managing Multiplayer Experience
With the spatial chat implementation, all we need is to see if it works! This project is set up to run locally with Netcode for GameObjects. If you would like to know more about the mechanism of how it works or to move it over to a future project, @Exhit-Game Developer developed a quick and thorough review. To try it out with our friends over the network, you would need to host the game on a server. With Unity Game Server Hosting, there are different options to try out, like Matchmaking and Lobby (shoutout to @CodeMonkey and @samyam for their excellent tutorials)! Now that it is established, you could either Build and Run the project to make another instance of the experience, or since it is implemented already, let’s do ParrelSync! At the top of the game engine, select ParrelSync and click on Clones Manager. Next, press Create new clone to duplicate the Unity editor to build the project by having another Unity editor window open and mirror the changes from the original project. After the duplication has been processed, then click on Open in New Editor.
Now we press play in both editors to see how the experience works! In one editor, you will click Host to initialize the Netcode, while in the other one, select Client to join the adventure. At this moment, you should be able to see two players in each other editors and hear your voice echo on the other end of the editor! And that is it; you have successfully implemented spatial chat into a multiplayer experience.
For more information on additional features to work on with the SDK, engage with our documentation page for API references and another getting started. In the spirit of sharing, you can head on over to YouTube for content about our other engine plugin – the streaming SDK. This allows you to either stream something into the experience or stream out to a large number audience. Lastly, Dolby.io is showing up next week at AWE US 2023, and I will be there speaking about spatial audio and chat, but if you don’t have tickets for the event, the talk is also available here!
