Send a Custom Video Using DeepAR.io and the Dolby.io Web SDK

The 3.7.0 release of the Dolby.io Communications SDK for Web introduces a new capability that allows you to send a custom video to remote participants. In this blog post we will use the DeepAR SDK to create a custom video with some special effects and inject it into a Dolby.io conference.

There are two simple steps to get this project running; first we need to load and initialize the DeepAR SDK into our web application. Then, when connecting to the conference, we will generate a video track from the DeepAR output and inject it into the Dolby.io conference.

A fully working demo project is available on GitHub.

Initialize the DeepAR SDK

Load the DeepAR SDK on the web page with the following script tag:

<script
    src="https://cdn.jsdelivr.net/npm/deepar/js/deepar.js"
    crossorigin="anonymous"></script>

Create a canvas element in your HTML DOM. This is where the DeepAR SDK will paint the video after applying an effect. We will also use this element to capture a video stream to send to the Dolby.io conference. We do not need to display this element on the page as the SDK will return it to us under the form of a participant stream, so we can hide it with some CSS.

<canvas id="local-video" style="display: none;"></canvas>

Connect to the DeepAR dashboard, create a project and get your DeepAR App Key. When creating a Web App in the dashboard, you will be asked to provide a domain name. Provide the one that will be used to host your web application. If you only intend to run the application locally, you can use any value.

We are now going to initialize the DeepAR SDK. This example will load the extra SDK dependencies (WASM file, inference models…) from the JSDelivr CDN since it is where we loaded the SDK from.

// Paste your DeepAR App Key in this variable
const deepArAppKey = 'Your DeepAR App Key';

// Specify where to download the DeepAR resources
const deepARRoot = 'https://cdn.jsdelivr.net/npm/deepar';

// Point to the canvas where the video will be painted
const canvasLocalVideo = document.getElementById('local-video');

// Initialize DeepAR SDK, follow instructions from the documentation:
// https://s3.eu-west-1.amazonaws.com/sdk.developer.deepar.ai/doc/web/classes/DeepAR.html#constructor
let deepAR = new DeepAR({
    licenseKey: deepArAppKey,
    canvas: canvasLocalVideo,
    deeparWasmPath: `${deepARRoot}/wasm/deepar.wasm`,
    callbacks: {
        onInitialize: () => {
            deepAR.startVideo(true);
        },
    },
    segmentationConfig: {
        modelPath: `${deepARRoot}/models/segmentation/segmentation-160x160-opt.bin`,
    },
});

deepAR.downloadFaceTrackingModel(`${deepARRoot}/models/face/models-68-extreme.bin`);

DeepAR provides you with a lot of free filters that you can use in your applications; effects are available in package 1, and package 2 in the effects folder. Add the .deepar effect files into a folder of your web application and create a variable to link the list of effects so that the application can switch in between them.

// DeepAR video effects
// https://docs.deepar.ai/deep-ar-studio/free-filter-pack/
// https://github.com/DeepARSDK/quickstart-web-js-npm
const effects = [
  './effects/8bitHearts.deepar',
  './effects/Elephant_Trunk.deepar',
  './effects/Emotion_Meter.deepar',
  './effects/Emotions_Exaggerator.deepar',
  './effects/Fire_Effect.deepar',
  './effects/Hope.deepar',
  './effects/Humanoid.deepar',
  './effects/MakeupLook.deepar',
  './effects/Neon_Devil_Horns.deepar',
  './effects/Ping_Pong.deepar',
  './effects/Snail.deepar',
  './effects/Split_View_Look.deepar',
  './effects/Stallone.deepar',
  './effects/Vendetta_Mask.deepar',
  './effects/burning_effect.deepar',
  './effects/flower_face.deepar',
  './effects/galaxy_background.deepar',
  './effects/viking_helmet.deepar',
];

// Index of the currently selected video effect
let currentEffectIdx = -1;

Add a button to your UI to trigger the switch of the effect.

<button id="btn-change-effect">Change effect</button>

And when the end user clicks it, the code will apply a new effect.

document.getElementById('btn-change-effect').onclick = () => {
    // Pick the next effect in the list
    currentEffectIdx = (currentEffectIdx + 1) % effects.length;
    const effect = effects[currentEffectIdx];
    // Apply the effect to the camera
    deepAR.switchEffect(0, 'slot', effect);
};

What we have now is the DeepAR SDK loaded and initialized, and a button that will allow us to switch from one effect to another. Now we need to send the video to the Dolby.io conference.

Connect to a Dolby.io conference

This blog post assumes that you already have an application up and running that can create and connect to a conference. If you do not, you can follow this getting started for Web SDK.

From the canvas element in the HTML DOM, we can use the captureStream function from JavaScript to generate a video stream at 25 frames per second and extract the video track.

// Get a video track out of a canvas
const track = canvasLocalVideo.captureStream(25).getVideoTracks()[0];

This blog post is focused on generating a custom video track based off of a canvas, but you can inject any MediaStreamTrack into the conference.

Then, modify the join request options to add the customVideoTrack property to send this video track as custom video for the participant:

const joinOptions = {
    constraints: {
        audio: true,
        video: true,
    },
    customVideoTrack: track,
};

// Join the conference
await VoxeetSDK.conference.join(conference, joinOptions);

When starting the video while already in the conference, you can follow the same technique, generate a video track from the canvas element, and send it to the conference using the start function.

document.getElementById('btn-start-video').onclick = async () => {
    try {
        // Get a video track out of a canvas at 25 frames per second
        const track = canvasLocalVideo.captureStream(25).getVideoTracks()[0];

        // Start sharing the video with the other participants
        await VoxeetSDK.video.local.start(track);
    } catch (error) {
        console.error(error)
    }
};

Run the application

Because this SDK is using WASM (Web Assembly) code, it must run from a web server; You will not be able to run your app simply by opening the HTML file in your web browser from the file system. This is a security restriction from your web browser. A quick solution if you want to test your application from your local machine is to use express. In your web application folder, initialize an NPM package and install express.

# Initialize the NPM package
npm init -y

# Install express
npm install express --save-dev

Create a file index.js and copy the following code to load express and open a web server that listens on port 3000.

const express = require('express')
const app = express()

app.use(express.static('.'))

app.listen(3000);

Run the web server using Node.JS.

node .

You can now open your web browser at https://localhost:3000.

This last screenshot is taken from the sample application available on GitHub.