In our previous update we introduced the concept of multisource streams and audio multiplexing within our Millicast API and Javascript SDK that enabled the features needed to build your own interactive Clubhouse application with the added benefit of video for the host/presenter.
With our latest Millicast 2.0 release we have deployed more advanced features that enable our customers to add multiple interactive real-time video tracks to their applications, complementing the existing interactive audio functionality.
The new projection feature allows the viewer to choose any of the audio and video sources they want to receive dynamically within a stream. This feature enables Millicast to be used as a massively distributed SFU so you can build your own meeting-like application like Zoom, Meet or Teams.
This article describes the interactive functionality in detail, and the docs are also available here: https://docs.dolby.io/streaming-apis/docs/interactivity-apis
Viewer events
In this new release we have added new events that can be received by the viewer application to detect:
- when a new source has been published within the stream
- when a source has been unpublished within the stream
- the simulcast/svc layer information for the published video tracks of each source
- VAD multiplexing, which was already available.
The docs are available here: https://docs.dolby.io/streaming-apis/docs/viewer-events
The events are delivered over the websocket connection established by the viewer application with the Millicast servers.
If you use the Millicast Javascript SDK, you can subscribe to the events when using the “view” command, specifying which events you want to receive:
https://gist.github.com/murillo128/77d6c8d6a860aedae6ac883d8543d897
Source and Layer Selection
So now that you have the ability to publish multiple sources in a stream and detect those events within the viewer, the next step is to specify the source you want to receive.
For each audio and video track you can select and switch sources from the Millicast server and then forward the selected media to the viewer.
This is accomplished using the “project” command. You can specify the “sourceId” that you want to project, use the “trackId” of the source (received on the “active” event”) for each track, and the “mid” of the transceiver object for the media you want to receive.
Here is an example of an “activate” event:
https://gist.github.com/murillo128/1c0d726b55e9fca13c86be9595430d8a
You can send a “project” command to start receiving either of the tracks from the source in one or more of the transceivers you have already setup on the peerconnection:
https://gist.github.com/murillo128/839c03189a796eb65f9e138a66ef6065
If you want to stop receiving media from one of the sources, you can either send a “project” command to replace the media with a different source, or use the “unproject” command to stop receiving media for that transceiver:
https://gist.github.com/murillo128/f45c5764309224a6b7ee6082efa1b552
By default, the Millicast server will choose which is the best Simulcast or SVC layer to forward to the viewer based on the bandwidth estimation calculated by the server.
https://www.youtube.com/watch?v=DZlsk7BQ5Cs
In addition to selecting the origin source for the media, it is also possible to choose the specific Simulcast and/or SVC layer for each video track delivered by the Millicast server. You can do that either by specifying the “layer” attribute on the “project” command or using the “select” command for the main video track:
https://gist.github.com/murillo128/3b8f7f5a390c1ee8364f1d51633c0602
The layer information available for each video source is provided periodically by the “layers” event as shown above. If you want to switch back to the automatic layer selection, you would just need to send a “project” or “select” command with empty layer information.
Note that Millicast will not limit the number of tracks that a viewer is able to receive, but will limit the maximum bitrate per viewer to a maximum of 4mbps across all media tracks. You should configure the Simulcast/SVC bitrate of all the sources carefully within your Applications so they can receive the desired amount of video tracks in the viewer session.
The docs are available here: https://docs.dolby.io/streaming-apis/docs/source-and-layer-selection
Dynamic tracks
The last piece of the puzzle is to be able to receive more than one audio or video track within the same viewer session.
With the audio multiplexing feature in our previous release, you could create multiple audio tracks when the viewing session was started. However, you couldn’t have multiple video tracks and multiplexed audio from the server. So there was an important feature missing: the ability to add new tracks on demand.
This is now possible with the “addRemoteTrack” command, which will perform a local renegotiation and create the “track” event with the added track and transceiver:
https://gist.github.com/murillo128/6d22fde4ba9dedfe554b0a0f1807b487
Once the transceiver has been assigned a mid value, you can use it with the “project” command to indicate the media source you want to receive from the Millicast server.
The docs are available here: https://docs.dolby.io/streaming-apis/docs/dynamic-viewer-tracks
Start building the future today
To start using these features, sign up for a free developer account and access the Millicast Dashboard to set up API access. Then view the documentation to get started at https://docs.dolby.io/streaming-apis/docs/interactivity-apis
