Developer

Using an API to Analyze and Extract Media Data

SUMMARY

The Dolby.io Media Analyze API allows you to extract audio data from media for insights into content, quality, and loudness.


The Dolby.io team has been speaking with a number of people about how they work with media and how they create media data. These folks have ranged in skill from beginner to advanced but share a common appreciation for high-quality audio. The beginner may still be interested in programmatic content classification, clipping detection, and loudness. The more advanced user may be trying to solve a very specific audio workflow problem such as evaluating the signal-to-noise ratio as a time series or extracting Musical key, instrument, and genre identification from a music file.

It is with this range of users and use cases in mind that the Media Analyze API exists.

What Does It Do?

The Analyze API takes media as input and programmatically generates audio data from the underlying signal, creating datasets that consist of the following information:

  • General media Info
  • Clipping sections
  • Loudness (time-series)
  • Bandwidth
  • Signal-to-noise ratio
  • Content classification (time-series)
  • Musical key, instrument, and genre identification

The API adheres to the principles of RESTful architecture, meaning it is language and platform agnostic. For example, sign up for a Dolby.io developer account to get your API Key and you can run something like this:

curl -X POST https://api.dolby.com/media/analyze 
  --header "x-api-key: $DOLBY_API_KEY" 
  --data '{
     "input": "https://archive.org/download/OrsonWellesMrBruns/381030_64kb.mp3"
  }'

You’ll get a response back with a job_id:

{"job_id":"b6ab4237-9109-4acb-82f4-5545ca8cd6fd"}

This is an asynchronous task meaning your audio is being programmatically analyzed on the Dolby.io cloud. When it’s ready, we’ll be able to pick up the results. You do that by passing the same job id back to the API.

curl -X GET https://api.dolby.com/media/analyze?job_id=$YOUR_JOB_ID 
  --header "x-api-key: $DOLBY_API_KEY"

We’ll get a JSON response to take a look at. What have we learned?

Check Basic Media Info

This section includes details about the container such as duration, kind, and size. Insight into the audio structure like bitrate, channels, codec, and sample rate. There are many tools that do this, but with the Analyze API you have a powerful option for mobile applications or platform integrations.

You get back a JSON response with audio and container attributes defined like the following:

        "media_info": {
            "container": {
                "kind": "mp4",
                "duration": 10801.645,
                "bitrate": 79674,
                "size": 107575636
            },
            "audio": {
                "codec": "aac",
                "channels": 2,
               "channel_order": "L R",
                "sample_rate": 44100,
                "duration": 10801.621223993765,
                "bitrate": 78286
            }
        }

Review Loudness Profile

To deliver content that conforms to broadcasting standards like A/85 and R.128 you may need to do some analysis of your media or user-generated content. If you are delivering to popular media platforms like Amazon, Apple, SoundCloud, Spotify, Vimeo, and Youtube you can prepare your content for the best results and get a straightforward pass/fail validation test.

If you want to know if your audio is within range, the JSON response like this may be useful:

"loudness": {
    "measured": -15.27,
    "range": 4.31,
    "gating_mode": "speech",
    "sample_peak": -0.0,
    "true_peak": 0.07,
    "time_series": [
                    [
                        0.0,
                        -120.0,
                        -4.23,
                        -4.22
                    ],
                    [
                        1.0,
                        -120.0,
                        -8.06,
                        -7.95
                    ],
                    ...
    ]

Identify Noise, Clipping, and Silence

To understand how easy it is to listen to your content, it can be useful to understand the level of noise present. If you are mixing content from many sources it can be important to understand if there is silence as a lead-in, at the end of a recording, or periodically placed markers throughout.

If there is a lot of silence or clipping you may need to make some audio cuts and run noise suppression to reduce static background noise:

"bandwidth": 11197
"noise": {
    "snr_average": 82.42,
    "level_average": -101.87
},
"clipping": {
    "num_sections": 0,
    "sections": []
},
"silence": {
  "processed_region": {
    "audio": {
      "silence": {
        "percentage": 5.3,
        "num_sections": 2,
        "sections": [
          {
            "section_id": "si_1",
            "start": 3.08,
            "duration": 2.08,
            "channels": [
              "ch_0"
            ]
          },
          {
            "section_id": "si_2",
            "start": 8.3,
            "duration": 3.2,
            "channels": [
              "ch_0",
              "ch_1"
            ]
          }
        ]
      }
    }
  }
}

Is the Dolby.io Analyze API for you?

We certainly hope so and if there is more data you are looking to learn about your media let us know. Our Analyze API tells you much more about your audio than just the metadata.

If you are ready to include analysis as part of your quality control process in a media workflow, a gating step for user-generated content, or to help search across an archive of media this API can help.

Pick your favorite client language and library and get started with the Analyze Media Quick Start guide which will walk you through the steps or check out this blog here which highlights how you can use Analyze for a Serverless Media Workflow.

Leave a Comment

Jayson DeLancey

Developer Relations

Jayson DeLancey leads the Developer Relations team for Dolby.io. With 20+ years of software development experience, he is inspired by the blend of creativity and technology he sees from our customers. He devotes himself to improving the everyday developer experiences so that developers can focus their attention on the fun parts of writing code.

Get Started

Drive real-time interactions and engagement with sub-second latency

We are more than just a streaming solutions provider; we are a technology partner helping you build a streaming ecosystem that meets your goals. Get started for free and as you grow, we offer aggressive volume discounts protecting your margins.

Developer Resources

Explore learning paths and helpful resources as you begin development with Dolby.io.

Copy link
Powered by Social Snap