Enabling Spatial Audio in Your Web Applications

Update 2022/01/21: we have now started our Open Beta program for Spatial Audio. In your Dolby.io dashboard, create a new application and turn on the Open Beta flag in the settings.

At Dolby.io, we aim to deliver the best experience for your real-time communication needs. We have introduced the Spatial Audio capability in the Dolby.io Communications APIs as the most recent step to make your end user’s experience better. This allows you to place conference participants in a 3D rendered audio scene and hear the participants from the given locations.

To learn more about this feature and everything you can do with it, check out our documentation. In this post I want to guide you, step by step, to take an existing web / JavaScript application, and turn on Spatial Audio. We are going to take an existing application, available on GitHub and enable Spatial Audio in a few simple steps. This is what the layout looks like; the participants are already placed around the local user. Once everybody joins the conference, when someone speaks, the spatial audio rendering will make it easier for your brain to find where the user’s video tile is located on the layout based on where the sound comes from. It also makes conversations more engaging.

The avatar images are made by photo3idea_studio and Freepik from www.flaticon.com.

In your Dolby.io dashboard, create a new application and turn on the Open Beta flag in the settings.

Get the source code of this sample application in the original folder and let’s get started. I wanted to make this web application as simple as possible so I decided to use some very traditional JavaScript frameworks, such as Jquery and Bootstrap, to avoid having to write too much CSS and JavaScript. I am also using Express to run a local web server.

The application is really simple. As people connect to the same conference, they are placed around the local participant. When someone speaks, the circle will illuminate around the avatar or video and the sound will come out from that position on the web page.

The sample application contains the following files:

• index.html – web page that you need to use to display the application
• styles.css – some simple styles for the application.
• actions.js – JavaScript used for actions such as turning on and off the camera or muting and unmuting the microphone
• videos.js – JavaScript that contains the event handlers for the video streams
• utils.js – Javascript that contains some utility functions
• server.js – Node.JS application that will start a small web server to deliver the web application locally

Follow the Getting Started guide for JavaScript if you are new to our platform.

First run of the sample application

Let’s start by running the current application without Spatial Audio. Since we are using Express to create a local web server, you first must install the package dependencies by running the command:

npm install

Create a file called .env in your application folder and set the consumer key and secret from your Dolby.io dashboard.

CONSUMER_KEY=<YOUR_CONSUMER_KEY_HERE>
CONSUMER_SECRET=<YOUR_CONSUMER_SECRET_HERE>

Run the web server using the npm command:

npm run start

Then you can access the application at https://localhost:8081

Initialization of Spatial Audio

Now that the application is running, to enable the Spatial Audio rendering, the very first action you need to do is to set the spatialAudio flag to true when joining the conference. We are going to work only in the file scripts.js.

const joinOptions = {
    constraints: {
        audio: true,
        video: true,
    },
    spatialAudio: true, // STEP 1 - Turn on Spatial Audio
};

Hint: Look for the following comment in the code to locate the joinOptions object:

// STEP 1 - Turn on Spatial Audio

Next, we need to describe how we want to define the spatial scene. In this function, we are going to consider the top left corner of the window as the origin point with coordinates { x: 0, y: 0, z: 0 }. As we move to the right our x will increase and as we move down our y will increase.

const setSpatialEnvironment = () => {
    // document.documentElement.clientWidth and document.documentElement.clientHeight give me the dimensions of the web page
    const scale   = { x: document.documentElement.clientWidth / 4, y: document.documentElement.clientHeight / 3, z: 1 };
    const forward = { x: 0, y: -1, z: 0 };
    const up      = { x: 0, y: 0,  z: 1 };
    const right   = { x: 1, y: 0,  z: 0 };

    VoxeetSDK.conference.setSpatialEnvironment(scale, forward, up, right);
};

Hint: Look for the following comment in the code to locate where to add this function:

// STEP 2 - Function to set the environment

Make sure to call this function right after joining the conference:

// Join the conference
await VoxeetSDK.conference.join(conference, joinOptions);

// Set the spatial audio scene
setSpatialEnvironment();

Hint: Look for the following comment in the code to locate the join function:

// STEP 3 - Set the spatial audio scene

Placing users in the spatial scene

In our sample application, each user is placed on the web page inside a div with the css class video-container and has a data attribute data-participant-id to make it easier to look for the location of that particular user. For that operation, we create a function setSpatialPosition that takes a participant object as parameter. We look for the DOM element that hosts that participant and gets the location of the DOM element on the page, and then tells the SDK to place that participant in the center of the DOM element.

const setSpatialPosition = (participant) => {
    // Look for the participant element
    const videoContainer = $(`[data-participant-id="${participant.id}"]`);
    if (!videoContainer.length) return;

    // Get the position of the UI element
    const elementPosition = videoContainer[0].getBoundingClientRect();

    // Get the position of the center of the UI element
    const spatialPosition = {
        x: elementPosition.x + (elementPosition.width / 2),
        y: elementPosition.y + (elementPosition.height / 2),
        z: 0,
    };

    VoxeetSDK.conference.setSpatialPosition(participant, spatialPosition);
};

Hint: Look for the following comment in the code to locate where to add this function:

// STEP 4 - Function to set the spatial placement for a user

On the participantUpdated event of the VoxeetSDK.conference object, when the user is connected, place that participant on the spatial scene. The participant will be muted by default until you place the participant in the audio scene.

if (isConnected(participant)) {
    $(`.user-container[data-external-id="${participant.info.externalId}"] .participant-name`).html(participant.info.name);
    setSpatialPosition(participant);
}

Hint: Look for the following comment in the code to locate the participantUpdated event:

// STEP 5 - Set the spatial placement for the user

Listen to the window resize event

When the end user resizes the window, the position of each UI element is changed so we have to listen to the event resize of the window object. Add the following code after joining the conference:

// Listen to the resize event
window.addEventListener('resize', onWindowResize);

Hint: Look for the following comment in the code to locate the join function:

// STEP 6 - Listen to the resize event

We need to write the function onWindowResize triggered on the resize event. Keep in mind that when resizing the window, the event will trigger A LOT. To prevent the function from triggering too often, we will use a mechanism with setTimeout and a “lock” to prevent the function to trigger more than once per second. In this function, we want to reset the dimensions of the spatial audio scene. Since the UI element that represents the local participant is at the center of the page, we need to move that participant to the center of the newly resized window. Then, because all the participants UI elements have moved, we need to call the setSpatialPosition for all of them.

var lockResizeEvent = false;

/**
 * Triggered when resizing the window.
 * The event will be processed with a maximum rate of once per second.
 */
const onWindowResize = () => {
    // Make sure the event processing is not "locked"
    if (lockResizeEvent) return;
    lockResizeEvent = true;

    // Use the setTimeout to wait a second before we process the resize event
    setTimeout(() => {
        lockResizeEvent = false;

        // Re-set the spatial audio scene dimensions
        setSpatialEnvironment();

        // Since the local participant UI element is at the center of the screen,
        // we need to update its position based on the new window size
        const localParticipant = $(`.user-container[data-participant-id="${VoxeetSDK.session.participant.id}"]`);
        if (localParticipant.length) {
            // Get the dimensions of the rectangle
            const clientRect = localParticipant[0].getBoundingClientRect();

            // Set the local participant at the center
            const top = (document.documentElement.clientHeight - clientRect.height ) / 2;
            const left = (document.documentElement.clientWidth - clientRect.width ) / 2;

            // Animate the UI element to its new position
            localParticipant.animate({ top: `${top}px`, left: `${left}px` }, 500);
        }

        // Now, for each participant, we need to update the spatial position
        [...VoxeetSDK.conference.participants].map((val) => {
            setSpatialPosition(val[1]);
        });
    }, 1000);
};

Hint: Look for the following comment in the code to locate where to add this function:

// STEP 7 - Handle the window resize event

You can find the completed version of this blog post in the final folder.

Deploy application

To invite someone to your conference and experience Spatial Audio, you can use a service such as ngrok to create a public URL from your local machine, or you can deploy the application on a cloud service and share the link.