Validating HTTP Live Streams

Hello. Welcome to this session on Validating HTTP Live Streams. My name is Eryk. I'm a media engineer at Apple. Let's get started. What are we doing when we validate a stream? We're trying to ensure the stream is correct. At least in terms of its structure.

Structural checks can't find all the potential problems, but it will eliminate a lot of them.

To be more precise, we first want the playlist and segments to be properly formatted. That is, is the software going to be able to parse the stream? Second, we have some additional requirements. For example, the HLS spec does not require I frame playlists. But we want to require them, because they improve the experience for the user.

Third, we want these streams to reflect established best practices.

Ultimately, this is all in service of making sure the stream plays well on your devices.

So let's talk about the specifications we have.

The most important is the HLS specification itself. If you've been working with HLS for any length of time, you've probably read this document a time or two.

It is an Internet Engineering Taskforce -- IETF, internet draft. You can find it just by searching on the web for "draft pantos".

The first version was published seven years ago. We updated two or three times a year. It details the structure of playlists. What kinds of tags exists, and their format. And explains the responsibilities for the clients and the server.

More recently we introduced a new document. The HLS Authoring Specification.

This is available in the reference library. We first published it last fall just for tvOS. For WWDC, we've updated it to cover iOS and OS X as well. This document tries to cover what constitutes well-structured HLS content.

The requirements in the authoring specification come from several sources. Some echo requirements from the HLS specification. For example, where discontinuities should be. What your playlist duration should be. Some have to do with device support. For example, that all your video needs to be H.264. Or that you can't have more than 60 frame per second content.

Others come from the desired user experience. For example, the I-Frame intervals that we expect. Or, the fact that we want you to support trick play.

A number are informed by the past seven years of our experience and our analysis. I'm talking about the recommendations we have around delivery. The bitrates we recommend. And our recommendations around privacy and security.

The requirements I've listed here are just some examples to give you a flavor of what's in the document. The specification itself is broken into categories. Like video encoding. Trick Play. Media segmentation. Delivery and so on.

Now of course we don't expect you to do this validation by hand. So let's look at the tools we provide. But first, you need to get the tools. Everything important about HLS can be found here at the developer streaming page, developer.apple.com/streaming.

Now if we scroll down a bit, down -- halfway down on the right, there is a link to the "Tools" download.

Now please note you will have to log in with a developer enabled Apple ID in order to access this page. But remember that developer accounts are free. Now once you've done that, you end up on the downloads page. The download itself is a disk image which contains an installer package. Now the screenshot here is a little out of date. You should see a newer disk image dated June 2016 or later. We do typically update the tools several times a year.

Now by the way, if you are viewing this during WWDC, the new version may not have been posted yet. After you've run the installer, you end up with sever tools. And I want to remind you that these tools run only on OS X.

The first two -- the validation are what I'll cover in the rest of this talk.

The next three tools are segmenters for files, streams, and subtitles. And I wanted to mention as a quick side that the subtitle segmenter is useful for breaking up the large web VTTT into a set of segmented web VTT's. The last two tools, Variant Playlist Creator is used to create master playlists.

And the last one is used to format timed metadata that you can then insert using the segmenters. Now all of these tools have manned pages. And you can pass any of these attached H option to get help.

But before we talk about the validation tools, I want to make a short pitch for mediafilesegmenter. It's simple to call, you just pass it a movie file.

It produces a media playlist and media segments. While this tool has a lot of options, it's not as complicated as it might seem. The options break into a few categories.

Options having to do with naming files. With structuring playlists.

How to do encryption and so on. Even if you don't use this introduction, the mediafilesegmenter can be a good cross check on whatever process you do use to segment.

So on to validation.

Media stream validator is the core tool.

You can call it with your master playlist.

And it produces output on the terminal and the json file. Now this json contains everything that was important in the output, and more data besides.

A couple of options that I want to call out. First of all you can adjust the path and the name of the json file. And you can adjust the timeout. Well, what's the timeout? By default, the validator times out. That is, it stops processing after five minutes. Now this may not be enough time to process a big video on demand stream. And for live content, while five minutes is fine for your initial checks, you'll want to do some checks over a much longer period, once you determine that the stream is basically correct.

Here's what the media stream validator output looks like.

So here we're calling media stream validator with URL.

And first it tells you the version number. And then it outputs some trace as it's downloading playlists and files.

And then it outputs information about every variant, rendition and I-Frame variant that it found. Now this can be hard to digest. So we created HLS report. This is a python script that takes the json and summarizes and condenses it.

And it produces an HTML file as output.

The name of that file is the same as the name of your json file, except the extension is changed.

And again, there are some options I want to call out. We can change the name of the HTML file.

The dash I or dash ID option is helpful when you want to match HTML output to portions of the json file. And I'll give you some more details about how that works when I go through the examples.

Now the verbose option is a very handy option that gives you a lot of information about every variant and rendition. Unfortunately, I don't have time to go into detail about that output. But I wanted to call it out as something that is available.

One quick aside here. You should validate your streams before you submit to the App Store.

And also, when you submit to the App Store, you should include a test stream URL in your notes field so that the reviewers have a stream that they can look at.

I'd like to run some demos. But rather than run all this on the screen and have you try to follow along and squint at tiny text, I've captured the output and I'll guide you through it. First, a little command line sample.

Run media stream validator. And notice I'm giving the name of the json file. And I've just got a fake URL here to give you the flavor.

Now we've seen the output before, so I'll skip that. But remember that this might take several minutes to run.

Now once it's completed, we can run HLS report. Passing at the json that media stream validator created.

If HLS report works well, we don't have any output. Just the HTML file.

So if we look in our current directory, we'll see there are just two files around. The json that media stream validator created. And the HTML file that HLS report created.

So let's take a look at some example reports.

First, I'm going to start with a playlist that has several common errors.

This is what a report looks like.

The first thing you want to examine is the stream type.

Now usually, this is VOD, or LIVE, or EVENT.

These reflect a three kind -- basic kinds of streams that HLS supports. But you could get the value "UNKNOWN". And this means that we weren't able to read the playlists. Or worse, you could get the value mixed. Now this can mean that you had some unknowns. That is, that some of the media playlists couldn't be read. Or worse, it could be that you actually have conflicting types. Like for example, some of your variants claim to be EVENT, and some claim to be LIVE.

Now the next item in the report is the variant overview table. It has one entry for each video variant.

Now if we happened to have our stream type as mixed, that will be an added column in this table that indicates the playlist type. If we look over to the right, we can see that there are more columns. And in particular, if it's a VOD we're going to have a percent processed column. Now if this is less than 100%, that means that the timeout wasn't long enough. Now if you have live content, you won't see the percent process. Instead you'll see two columns. Average segment count, and average playlist duration.

Now the next thing you want to check is the percent difference column. This shows you how much greater the measured bitrate is than the value that was declared in the master playlist.

Now these values are way too high. Notice that variant one has a -- is 40% over what it should be. Now these values should be in the range of minus 10% to positive 10%.

If the value here is positive, that means your declared value should be increased. And if the value is negative, your declared value should probably be decreased. Now notice we've actually got two of these. Because in our playlist we declared both the bandwidth and the average bandwidth. We are able to compare the declared and measured values for both.

Now sometimes looking at this in terms of percentage can be a little hard to see what the real difference is. So if you add the -- pl option, meaning playlist, to your HLS report command line, you'll see two new columns.

Playlist rate. And playlist average rate.

So with this, we can see that for example, for variant one, the declared value is 392 kilobits per second. But the measured value was 551 kilobits per second. Now let's switch back to a version without the -p option and scroll down a bit.

There's another table, the rendition overview.

In this master playlist we have audio as a separate stream. That makes it a rendition, and so it gets reported here.

Following this, we have our must fix issues. Now must fix issues are things which are essential for correct playback. Or things which we consider so important that we made them hard requirements.

You see item #2 and #4 here are what we were just talking about. The fact that the peak and average bit rates, that the declared and measured values don't match up.

Now item 3 we'll address in a later example.

Scrolling down farther, we now have our should fix issues. Should fix issues are things which we strongly encourage you to do. Now these are followed lastly by the report information. The report information is so that we can backtrack to which version of media stream validator and HLS report was used. This is important if you pass us in a bug report, the output from HLS report. Or if you're even going back and looking at something after you've downloaded a more recent version of the tools.

Now item 5 here is telling us that we don't have any captions or subtitles in our playlist. We'll fix this in our next example.

Let's look at item number 7.

So item number 7 says that our default video variant is not what we suggest.

And it's telling us about variant #8.

So let's scroll back up to the variant overview. And we can see that item #8 has the item number highlighted. That's how we can tell it's the default variant. Now remember the default variant is the variant that the client's going to play initially.

It's the first variant in the master play list. And notice that this one, the max rate is 8 MB per second. That's way too high as a default.

Okay. Those are some common problems that you might have. In our second example, we've taken the first example and fixed the problems I pointed out.

We've adjusted the declared values, so now the percent difference is nearly zero.

We've fixed the default variant to be something more reasonable -- variant #4 in this case -- by moving a different media play list to the first position in the master playlist.

And if we look over on the right-hand side, we fixed our timeout, so we're now processing the entire playlist.

Now we had a should fix item about captions. We fixed that by adding a subtitle playlist.

Now notice as a subtitle's rendition -- rendition #12 -- doesn't have a max or average rate.

That's because the subtitle playlist is not processed. And we put an n/a over in the percent processed column to remind us that this is in process.

But we still have one big must fix, and that's no I-Frame playlist.

So how does the report change when we add I-Frame playlist in? The first part looks the same, but let's scroll down.

Now, in addition to the other overview tables, there's an I-Frame variant overview. Now notice we're a little off on our percent difference values here. But these are well within the range. Remember, that's +/- 10%.

Now let's talk about these two columns over here. Scaled average and multiplier. Scaled average is the affective worst case average data rate of the I-Frame playlist.

Yeah, that's kind of a mouthful. Remember that the measured, average bit rate represents the data rate at a 1 X playback speed. But the whole point of I-Frames of Trick Play is to play back at greater than 1 X speed. Now the multiplier tells us the lowest playback rate at which we will hit that worst case value.

Higher rates are going to be no worse, because we will drop frames in order to keep the actual displayed frame rate at a constant value. What you should do with these values is compare the scaled average data rate here, against the normal variant data rate that has the same resolution.

These should be close in value, or the I-Frame variant should be lower in value. Otherwise, Trick Play could cause a drop in resolution.

Now in this report, I've also got another new column. That's because I added the -- id option to HLS report. Now as I mentioned before, this helps you find the same data in the json file.

So let's look for this first one, data ID 16. Which is our lowest resolution I-Frame playlist. Let's look for that in the json.

So here's an abbreviated text view of the json. The top level of the json is a dictionary. And we're interested in the variance value. Which is an array of dictionaries.

Now one of those dictionaries will have data ID 16.

And I want to note here that these are not in order by data ID.

Now this one with data ID 16, that's the json variant entry whose data matches the row in the report. That is the lowest I-Frame variant.

Now, I've skipped a lot of the data. There's much more information than this slide shows.

Use your favorite tool for digging around json files, or write yourself some special purpose tools. Now, these json's can get very large. It's not uncommon for them to be over 100,000 lines in size.

For our last example, we took that playlist and we added multiple kinds of audio to it. So the top looks pretty similar. Let's scroll down and look at our renditions. Now, rather than just that single, stereo audio, we've added two multi-channel audio renditions as well. One is AC-3 that is Dolby Digital. That one's called the aud2.

And one is EC-3, Dolby Digital Plus.

One we call aud3. So let's go up and look at our variance section.

Now, notice that because AC-3 and EC-3 are not substitutable, and neither are they substitutable with AAC, each one has its own default variant.

That is, once we choose aud2, the AC-3, we will default to variant #12. Because that variant is the first one in the aud2 group that appears in the master playlist.

Similarly, if we chose EC-3, we would default to variant #20.

Now notice there's a bitrate difference between the aud2 and aud3 variance. For example, variant #9 and variant #17. This is due to the different bitrates of the audio. The bitrate you'll see here for the variant is the combined bitrate including the renditions.

Okay. That's it for my examples. Where can you find some examples of your own to look at? Well, there are three examples pointed to off of the streaming page. If you go to developer.apple.com/streaming, there's a pointer to the examples page. And you could dig the URLs out of this page.

Or, you could just read them on this slide.

The first two will have complaints when you validate. That's because these streams were created a long time ago, and we deliberately decided not to upgrade them. But that third variant is a new one, and should be almost perfect.

All right, it's time to wrap up.

Here are the only three things I absolutely want you to remember.

Obey the authoring spec. Validate, validate, validate your streams. And include a test stream URL when you submit your app. For more information, go to the WWDC site, and use this session's number, 510. There's a lot of cool new stuff this year in HTTP live streaming. So I encourage you to attend or watch the talk, What's New in HTTP Live Streaming. Thank you, and have a great conference.