Measuring and Optimizing HLS Performance

Hello. Good morning, everyone. And welcome to this year's session on HTP Light Streaming.

My name is Emil Andriescu. Today's talk is about measuring and optimizing HLS performance.

First, let's reflect for a second on why we should care and why it is essential for application. Let's pretend it's Saturday night, you're in your favorite spot on the couch, you've skillfully browsed through all the reviews, title is set, popcorn is ready, and you eagerly tap play when this happens.

Faced with this mysteriously and never-ending animation, you ask yourself, what could be worse.

Let's face it, you know it, it's a playback error. But what do customers really expect from HTP Light Streaming? Well, they expect high-definition image, high fidelity sound, and instant media response when they tap play.

Yet, streaming applications over the internet are always at the mercy of the network, so how do we reconcile? Well, HTP Light Streaming was designed to address this, that is to provide the best possible uninterrupted streaming experience in an unpredictable network environment. So why are we here? Well, there's more. Over time, HLS has evolved into a more dynamic ecosystem, supporting new offering features such as I-frame playback, new media formats, and of course new codecs. At the same time, we're constantly adding powerful iOS, tvOS, and macOS APIs such as you can tune and adjust playback to your target audience and provide a much richer user experience.

Delivery patterns and transport protocols, they are also evolving, so it is important to look at your server side performance in connection to how content is being consumed, either on a mobile device or in the living room. Given all these options, how can you be sure that you are providing the best possible user experience to your audience? Well, the first step is to understand and quantify the user experience in conjunction to changes that you make to your content, application, or delivery.

This is an area where we believe it is imperative to measure rather than to guess which configuration is optimal.

What is this session about? First, we want to establish a common language for discussing streaming quality of service. Second, we want to discuss how to objectively measure your application streaming performance.

Third, we want to help you identify and solve some of the problems that impair streaming quality of service.

And finally, we want to get those master playlists right. This is because many of the problems and issues that we see with streaming quality are actually rooted in the authoring of the master playlist.

Before going into detail, let's begin with a brief overview of an HLS playback session.

As you'd expect, it begins with the download of a master playlist. Once the playlist is passed by AV Player, it knows what content it refers to. In this case, we have two bitrates, 1 megabit and 2 megabit. AV Player will pick one of these, will go ahead and download a media playlist together with additional artifacts such as keys, and then continue to download media segments until the buffer level is sufficient for playback. When that happens, the AV Player item will communicate a prediction of playability by setting the is playback likely to keep up property to true. If you've preset the AV Player Rate to 1, so you're using the Autoplay feature of a AV Player, the player will go ahead and start playback immediately. We call this Time Interval Startup Time.

From this point on, the wall clock, also known as real time, and the player item time base will advance at the same speed, with one condition, which is that content must arrive at an equivalent or faster rate than that which AV Player is consuming. If that's not the case, AV Player will try and switch down to the 1 megabit here. If network still cannot keep up with real time at 1 megabit, well the buffer will eventually run dry, and AV Player has no choice here, it needs to stop playback, event which we call a stall.

The player will remain in this state not only until data starts flowing again, but up until there's a sufficient level of buffer for the player item to trigger another positive playability prediction.

After that, playback can continue normally.

Now let's discuss about quantifying the user experience for such a session.

We do that by defining a set of Key Performance Indicators or KPIs. We picked five of them that we believe are most representative for HTP Light Streaming.

One question you may ask is how much time do my users spend waiting for playback to start. Is it one second? Is it five seconds or maybe 30 seconds? This is an essential point in terms of user experience.

Further, playback stalls, like the one we just saw, they are disruptive to the user. We care both about how often they occur, but maybe more importantly, how long does it take to recover from a stall? Yet, the best strategy without knowledge of the future to not stall is to deliver content at the lowest available bitrate. But of course that's not what we want. We want to deliver the best audio and video quality while still not stalling. So there's a clear tradeoff between risk of stalling and media quality, right. For that, we need another measure of the overall media quality for a session.

And finally, playback errors. We talked about that. They are more disruptive than stalls, right. What can we do to track playback errors? Okay. Let's begin with startup time.

There are multiple APIs that you can use to obtain or compute startup time. First, don't use the AV Player status changing to ready to play. That doesn't tell you that playback is going to start. However, if you are using Autoplay, so you're setting the rate, the player's rate in advance, you can use the AV Player item status changing to ready to play or the AV Player item is playback likely to keep out changing to true. These are observable properties.

When that happens, you know that playback is about to start, but there might be a few milliseconds before playback actually starts. So what we recommend is to either use the AV Player time control status changing to playing or to track the player item time base, and there's a notification that allows you to do that. AV Player relies on heuristics to avoid stalls, but we know it, sometimes they're unavoidable.

You can monitor as stalls occur by registering to the AV Player item playback stall notification. The suggestion here is to count the occurrence of stalls. Of course, if you want to compare and aggregate stall behavior across sessions of different duration, then you need to normalize this. How do you do that? Well, we recommend that you use the total duration watched and compute the stall rate in terms of stalls per unit of time watched, such as stalls per hour. A stall of 30 seconds is much worse to the user than a stall of one second. This is why we also care about rebuffering time or stall duration.

By measuring the time interval between playback stalled notification and when the player item time base changes back to 1, you can compute an accurate stall duration. Again, the total duration can be normalized using the duration watched of the session. Well, you might be wondering at this point, how do I compute the duration watched of a session? And the answer is, through the Access Log. Let's see how we do that.

So this is a snippet of code.

First, we get a reference to the Access Log from the player item.

We iterate through the events in the Access Log, and we simply sum up each events duration watch. And there you have it. We computed a total duration watch for a session. And now you may be wondering, well what is this event? What is an event in the access log mean? Well, for that let's look at how AV Player Item Access Log works.

So the AV Player Item Access Log provides a history of your session. It is initially null, but as playback occurs, you're going to receive an AV Player Item New Access Log Entry notification, and by that time, you'll have an Access Log. You'll see that events in the Access Log contain information on various areas such as the current variant URL, the current bitrate, duration watch, number of stalls, and so on.

These values are initially by convention initialized to a negative value or null. As playback occurs, they are updated with actual measurement data and the actual variant URL that you're playing. There are two cases in which we will add new events to the Access Log, and that is variant switch, like in this case, or a playback seek. But before a new event is added, the old one becomes immutable, and then we add the new event. Now, keep in mind that while these values here are constantly updating as playback occurs, so the values in the last event, they are not observable properties.

We also mentioned that we care about media quality. How do we compute that? A way to measure if the user is presented with the best possible media quality is of course to look at the video bitrate being served. Here we don't really care about the startup time or the stall duration, so let's remove those. So we're left with the playback state. In this example, we see that we played for a longer time on the 2 megabit variant and less time at 1 megabit. By time weighting each bitrate, we can obtain a single value of video quality that we can compare across sessions.

We call this measure a Time-Weighted Indicated Bitrate, and computing it is just as simple as with the total duration.

Once again, we get a reference to the Player Items Access Log. We iterate through the events in the log. We compute the time weight of each event with respect to the total duration watch we computed earlier, and finally, we sum up the weighted bitrate value. Now keep in mind that some of these properties may not be initialized, so do the appropriate checks in your code.

Another event which you must absolutely track is of course playback failure. To do that, you observe the AV Player item status.

If the value ever changes to false, it means AV Player encountered an unrecoverable error.

A good way to transform this observation into a KPI? Well, one way to do it is to look at the percentage of failed sessions with respect to total sessions, but there might be other ways to do it. One thing I want to stress here is that not all errors in your stream may be fatal. Some may impact media quality while some might not even be perceivable by the user. But nonetheless, if there are errors, they convey that there is an issue with your stream. So how do I get more insights on the stream, right, what happened? And the answer is from the Player Item Error Log.

The AV Player Item Error Log. The Error Log conveys failures with varying degrees of user impact. It works in a similar fashion as the Access Log except that events represent error rather than player access states.

They cover various areas, such as delivery issues, network issues, content authoring errors, and so on. For instance, they can give you an insight on why a stall occurred, such as no response for a media file for about ten seconds. So we talked about startup time that you can track for every session. We encourage you to take a look at the distribution of startup times for your application. We also talked about stall occurrence and stall duration.

We mentioned that Time-Weighted Indicated Bitrate is a good indication of experienced media quality across a session, and finally, you probably want to keep the percentage of failed sessions as low as possible.

Keep in mind that not all KPIs are comparable across sessions. One example of that is that AV Player foundation uses the AV Player layer size on the screen to evaluate illegible variants for HLS. So for instance, if you've got 10 ATP content, it will probably not be displayed on a 200 pixel view, but it doesn't mean the user experienced poor image quality. What to do then? We recommend that you gather additional context information along with your streaming metrics. This will allow you to later partition your playback sessions in classes that make sense for your application.

Sample code for this section is available on the Apple developer website as part of the HLS catalogue sample.

Now, please let me welcome Zhenheng Li, who will talk to you about ways to improve HLS performance. Thank you. Thank you, Emil.

Hello, everyone. My name is Zhenheng. We have discussed all the KPIs that our users care the most. In this part of talk, let's focus on ways to improve these APIs. We will look deeper in three areas. One, how to reduce the startup time.

Two, how to investigate and avoid stalls.

Three, how to investigate and avoid errors. Let's get started.

So what can delay start of playback? Here is an example of the operations from the user clicks play until the video start to play back.

The application create every asset and start inspection of the asset to find out durations and awardable media options of the asset.

It takes a few round trip between the device and the content server to download the master playlist and playlist.

After that, the application create AV Player and AV Player Item.

Buffering starts.

Oftentime, buffering is interrupted, content is encrypted.

It takes a few round trips between the device and to fetch the decryption keys.

Once the keys are fetched, buffering resumes.

However, it may be interrupted again. Let's say the application offers a feature, resumes from the previously watched point.

Application sets a sic time, a set time on the player on behalf of the user.

Every player discard the existing buffer and start download from a new location. Segment 100.

Again, it maybe interrupted. Users has a language preference setting in the application.

She or he prefers Spanish audio.

Thus, application sets media selection on the player item, existing audio buffer being discarded, player start downloading from a different language variant. In a few seconds later, player item notifies playback is like to keep up, application sets a rate.

Playback starts, and it continues. All this time, user is waiting.

So as we can see, it does take a few time-consuming operations to start up, run a trip between the device and the content server and the key servers. Round trip times between AV Player and applications, oftentimes these two sit at different processes.

So how the application measures the time cost and startup time? It may measure the time spent between the API calls and the player and the Player Item status change notifications. Every player item also offers startup time in the Access Log. This time is measured by the AV Player item, represents the time for buffering only.

It's measured from the start media downloading until the first playback is selected to keep up.

So our user wants the video to start fast, in at most a few seconds. There are ways to achieve that. One option, we can move some operations to a different stage before the user clicks play.

For example, AV Asset creation and inspection can be moved out. Key fetching can be moved out.

Thus, when the users starts a video playback, there is less waiting time. So where do we move those operations to? While your user is viewing the video catalogue or video info, it's a good time to create an inspect AV Asset before the user decides to play. Now, last year we had introduced AV Content Key Session API.

This new API decouples the media load from key fetching. It gives the application total control on key management. It offers ways to optimize key fetching, such as bundling up multiple key requests back to the key server.

If you happen to adopt AV Content Key Session, spending a few hours of engineering hours, your user will notice a faster startup time.

So we have moved the AV Asset creation and key fetching out of startup time. Now what's left is mainly the AV Player Item buffering time and the communication time from AV Player and your application.

Oftentime, app may be able to avoid buffering, such as due to or due to media options. We can even try to reduce the round trip time between the player and the application.

Thus the startup is further reduced.

Let's take a look. When you create AV Player Item, if you know where your user is intending to start the playback, set the current time on player item. If you know what are the media options such as which language to download for playback, set that as well before you set the AV Player item onto the player.

Same with the AV plyer. As soon as the user click play, set rate before the start downloading for the AV Player Item. Thus, the playback will start automatically as soon as Player Item has enough to play back.

In summary, set up AV Player before buffering.

Set AV Player rate before setting the player item onto the player. A lot of application offers a feature to allow the user choose multiple videos and play one after another, such as binge watching TV episodes.

We have seen implementation such as one player and one player item per video. There's always a startup buffering time for each new video. You may reduce that buffering time for the new video by AV Queue Player.

Create multiple player items, include them all on the play queue. While the player is playing the current item, when the media download finishes for the current item, player will start downloading for the next one while the current one is still playing.

Thus, the next player item will start playback as soon as current event play to the end. So do use AV Queue Player to play multiple items and enqueue second AV Player Item well in advance.

So what's left now? Buffering time.

First, what determine network buffering time. Four factors.

The choice of your variant, the content bitrate, your playlist target duration, and of course, last, the network bandwidth. Let's take a look a few examples of buffering time.

First, it's a simple master playlist. It specifies an ATP video at about 5 mbps. Let's assume the network bandwidth is around 6 mbps. Our target duration is 10 seconds.

In most of the cases, player item buffers one segment before it notifies playback it like to keep up. However, the same master playlist, almost the same network condition, the user may observe slower startup.

The reason is, remember, the network bandwidths change, and the content bitrate also change. In this case, there are a few segments take longer to download. Thus, it takes longer to start.

To solve this problem, offering a variant with lower bitrate may help.

Player may decide to switch down and start up sooner.

When all other information is absent, the first listed variant will be your startup variant. So in this example, same two variants. The lower bitrate is listed first with same network condition. Player will start faster, start up faster and also switch up pretty quickly given the network bandwidth is sufficient for playback.

In summary, to reduce network buffering time, make a wise choice of initial variant.

Lower content bitrate means shorter buffering time, but it is a tradeoff of video quality.

If you are offering multiple media formats such as HDR and SDR videos or stereo audio and multiple-channel audios, make sure the initial variant for each media format are on similar level of bitrate so your user will have a similar experience regardless what kind of viewing setup they have. That's all about reduce startup time. Our video has started.

Next, let's talk about stalls.

To be really clear, stalls can happen, especially when the network bandwidth is really low. So in this part of talk, let's focus on how to investigate stalls and how to improve or avoid stalls. How the application investigate stalls. The application should be listening to the stall notification at all time.

And the application should be also checking the AV Player status such as is playback likely to keep up. AV Player Item also offers Error Log and Access Logs. The application should be listening to an exam those logs when the stall happens. Next, let's take a look two stall examples.

First, stall notification has been received by the application. The application should have received the Error Log as well.

The error comments give you detailed information on what has happened. In this case, it says media file not received in 15 seconds.

Application checks Access Log to find out what the AV Player was playing at the moment when the stall happened.

It tells you detailed information such as the player was playing what content and such URI.

The indicated bitrate is the content bitrate. In this case, 36 mbps, and that is a content.

An observed bitrate is the current network bandwidth. In this case is 2.8 mbps. It's obviously due to the network bandwidth can't catch up with the content bitrate. So to deal with variable networks, remember to provide a full set of bitrate. Remember some of your users may have a slower network connection, or your user may be on the go, such as on cellular while viewing the video.

If you're offering multiple video, multiple media formats, each codec combination needs it's own set of tiers.

Not all stalls are due to network condition.

Let's look at this one.

Stall happened, Error Log tells you a different story this time.

It says playlist file unchanged for two consecutive reads.

If you check the Access Log at the time, player was playing live.

They indicated the bitrate is rather low. The content is about 400K, and the network bandwidth is 3.7 mbps. This look like a content delivery issue.

So to reduce or to avoid stalls due to content delivery, content server and CDN must deliver media files, segments, keys without any delay.

Update live playlist at least every target duration. The CDN must be configured to deliver most recent playlist to avoid stale playlists. Synchronized discontinuity sequence number between playlist.

Indicate server-side failure clearly using right HTTP status code.

That's all about stall. What about error? How do we investigate errors? There are a few ways.

We have Error Log and Access Log from AV Player Item.

We also have error property from every player and player item that the application can observe.

In addition, we have some media validation tools for you to detect the content issue. Let's look at them one by one. AV Player Item Error Log, they have talked a little bit about in this one.

This type of Error Log is an indication that there is a problem with network or content format. However, they are not always fatal. When the error is indicated, playback may be perfectly fine at that moment.

However, the application showed the check in the error comments to find out details, such as this one.

We have seen it before, media file not received in 15 seconds. So it's an indication that your user may have observed or will observe stalls.

Now next one is HTTP error, it says file not found. This an indication of a content delivery issue.

The user may observe audio loss, video loss, or both. specified bandwidth for variant. Now this is an interesting one. It's an indication of a stall risk. However, the playback may be perfectly fine when the error is indicated. It means some of the segments bitrate is higher than what is specified in the master playlist. Last example, crypto format error, unsupported crypto format. This may be an indication of a failure, a playback failure.

All this error message and a few more that are not talked about here are very helpful when we have AV Player and Player Item errors.

Let's take a look.

The application should be observing AV Player Item status and AV Player Item error property to find out this type of error.

These errors are fatal errors.

When the error indicated playback has been terminated already, so what should we do? How do we find out the cause? Here is example.

The application is observing player item status when the status changed to failed.

Application go off to check the AV Player error properties as well as the Error Log from the AV Player Item.

Here is the error property from the player item. It provides some useful information.

Error code from AV foundation error domain. It also provides some hint, go off and check the Error Log from AV Player Item.

So corresponding AV Player Error Log gives you much more details.

It tells you on this data and the time and what URI with what type of error. So in this case, it's unsupported crypto format.

It also tells you what type of network interface the device was on when the error happens. Next type of error, HDCP.

If you are offering content that requires HDCP protection, your application should be observing this long property name, property. It's output obscured due to insufficient external protection.

The value of this property changes to two means three things.

Current item requires external protection. Device does not meet the protection level. User will observe or is already observing video loss, like through for example.

To avoid this issue, your master playlist should offer at least one variant that does not require HDCP for fallback. Remember, not all your users has the viewing setup that is HDCP capable.

App user interface should reflect the property change to timely hint the user.

A lot of playback issues are introduced by content authoring such as audio and video out of syncope or glitches while switching.

In addition to the error investigation and handling that we have talked about, we would encourage you to use our media stream validator too, which is available on the developer website.

That's all I want to talk about it today.

Now let's welcome my colleague, Eryk Vershen, to talk about how to author the master playlist the correct way. Thank you.

Thanks, Zhenheng. My name's Eryk Vershen. I'm an engineer working on HLS Tools. We've spoken about how to measure your performance and also how to address many of those concerns. However, one of the key elements to successful and error-free playback experience is to ensure that your master playlist is authored correctly. The master playlist is what allows the player to make intelligent decisions both before and during playback. So getting it right is critical.

There we go. That's my advice. No, I'm just kidding. I think I need to give you a little more background to understand what Roger meant.

We want you to put all of the encoding options you have into your master playlist and to describe them as completely as possible.

Let's pretend you're asking me questions. This is the crucial question and the main thing you have to get right.

Now, first you have to remember that just because a master playlist works doesn't mean it's right. I've actually seen master playlists that look remarkably like this. This is technically legal, and it's next to useless. I say, okay, well how about this one? It has a few more variants.

Well, it's a little bit better, but it's still terrible. Can we even play this? What codec is it using? Is it HDR? Is it 60 fps? You need to tell us everything. We want you to tell us everything. For example, average bandwidth. Average bandwidth enables us to make better decisions about which variant to switch to.

It's a better predictor or whether we'll be able to keep up with a stream.

Codecs is what enables us to filter out things that we can't play, and resolution allows us to make good decisions about which variant to choose.

Remember, we don't read the media playlists or the media segments until we have to. So you need to tell us things ahead of time in your master playlist.

So here's a sample of a simple master playlist. This playlist allows the player to adapt to bandwidth changes and make good choices about which variant to use.

Now, everything that we've done here is invisible to the user. It just makes the stream play better.

Okay. Let's look at a common problem. Your stream plays, but you're not seeing any images in fast forward, or your not seeing a thumbnail in the scrubber bar.

Here's the Apple TV scrubber bar.

You can see how long your content is. You can see where you are in the content, where you want to go.

Now, in order to get that thumbnail image, you need to give us an I-frame playlist, and the I-frame playlist is also what allows us to provide images in fast forward and reverse playback on your iPad or your iPhone. Now in order to talk about I-frame playlist, we first need to talk just for a moment about normal video.

Now, here's a way of visualizing regular video segments in HLS. Each segment has content for a number of frames, so it has a duration in frames, and it has a particular average bitrate, and that bitrate varies from segment to segment. Now, because of compression techniques, most frames in a video can only be decoded relative to other frames.

But I-frames, the I stands for intercoded frames, these are frames that are independently decodable, and they're the base frames that allow everything else to be decoded.

Now, as I've shown you here, you might have more than one I-frame in a single segment, and the I-frames need not be in a, occur at regular intervals.

An I-frame playlist is a playlist which just points to the I-frame content, that is only the I-frame data will be downloaded. And when we talk about the duration of an I-frame, we always mean the time from that I-frame till the next I-frame.

Now, this particular set of I-frames, I've shown as extracted from my normal content, but you can also make what we call a high-density I-frame playlist.

This isn't something just extracted from your normal content. Instead, you make it deliberately with more evenly spaced I-frames. This will allow us to work better. It allows us to give a much smoother result when you're fast forwarding.

Now, here I'm showing you a master playlist without I-frame playlist added.

Now, notice that the I-frame playlist has almost exactly the same tags as the normal playlist. The only difference is the I-frame playlist does not support the frame rate attribute because it doesn't make any sense in that context.

Now, a good test for your I-frame playlist is to try and play it directly. That is, take the URI of your I-frame playlist and paste it into Safari. It should play at 1X speed, and you should see the I-frames displayed one after another in a slowly changing sequence.

Now also I want to point out the difference in the bitrate.

Notice that the I-frame bitrate is much lower than the normal bitrate. That should always be the case. Now, speaking of bitrates, we've defined how to compute the peak bitrate in the HLS specification.

Make sure you do it that way. Otherwise, you may get that segment exceeds playlist, exceeds specified bandwidth error. Now, we're going to move away from video and talk about audio for a little bit. Now the most common question is how do I support multiple languages? Here's what the interface looks like. I've got a list of languages, and the user can select one.

And here's a sample playlist. Now, notice what we did is we've added an audio tag, sorry, an audio attribute onto each of our video variants, and we've added a group, the media tags with group ID's. The group ID is simply a tag that allows you to associate the audio renditions with the video variants. So notice there are a number of differences between the two audio renditions. Just as with your variants, we want you to tell us as much as you can about your media.

Now, there are two attributes that people tend to have trouble with on the media tags, and that's Default and Autoselect.

Okay. So Autoselect says that the media selection code in the player is allowed to choose this rendition without any special input from the user. Most of the time, you want Autoselect set to yes.

If you don't set this, the user's going to have to make an explicit choice to get that rendition.

The default on the other hand is what to pick when the user has not given a preferred language.

Generally this should be the original language of your video, and the default must be autoselectable because the system has to do the choosing. Now, this default has nothing to do with the default video variant. This is the default within the group of renditions.

So, okay, great. I've got multiple language, but I'd really like to have some multichannel audio. I've got 5.1.

Okay. Well the first thing to remember is not all devices can play multichannel audio. So you want to also provide the user with a stereo option. And you should think of this always as filling out a matrix. You need to have every format having every language.

You may say, well, I don't have a multichannel original for my French. I don't have a 5.1 French. In that case what you should do is put stereo, your stereo content in that group instead.

You need to have something in every slot of this matrix. So let's see a sample playlist again. This one is just like the previous example except I've changed the group ID, and remember that's perfectly fine because the group ID just serves to connect the audio renditions with the video variants. Now, here we had the multichannel group, and then I've set this up with French as stereo, so you can see how that's done. What you need to do is make sure that the codecs tag indicates all the different codecs that can occur within that rendition group. Now, notice that we had to duplicate our video variant. So now we've got two entries, one pointing to one audio group, and the other pointing to the other audio group. And you'll see this kind of duplication again in later slides.

Well let's say rather than 5.1, I've got several audio bitrates. I've got some high bitrate audio, and I know I need to provide a low bitrate for some users. So in terms of the renditions, this is similar to what we had before. We still got a matrix. We want to fill it out with every language for every bitrate.

And since these are both AAC, they're considered the same format. So if I also want to have another format, all I do is extend that matrix. Now, I want to mention that I've been saying language for convenience, but you should remember that it's the name attribute which is the unique attribute, not the language attribute. Now, in this playlist I'm not going to show you the media tags. I'm just going to show you the video variants with their audio group names. Now you want your low bitrate video associated with your low bitrate audio, and you want you high bitrate video associated with your high bitrate audio.

And you'll always want to split this up like this. Don't do a situation where you have a complete set of video variants associated with your low bitrate audio and a complete set of variants associated with your high bitrate audio. Because if you do that, you can be at a high video bitrate and be bouncing between high and low audio bitrates. Now, here I've added in the AC3 content. Notice that again we had to duplicate our video variant entries, but they point to the same video playlist.

Now, notice also that the bitrate on the video variants changes. Remember that's because the video, I'm sorry, the bitrate associated with the video variant is the bitrate of the video itself plus any associated renditions. Now, let's go back to the video for just a second because I want to have multiple video formats. I like to have HEVC, so I can have better quality at the same bitrate or I'd like to have Dolby Vision so I can have some HDR content. Again, we're kind of filling out a matrix. In this case, no matter which video format we choose, we want to end up with a reasonable set of variants. So the rows here are tiers based on quality, and we want to fill out the matrix with a variant in each tier in each format.

Now, we don't have to necessarily fill out the higher portions of the tiers on our older formats. You can skimp a little bit there. But similar to audio, not every device supports things like Dolby Vision, so you want to provide an H.264 variant as a fallback.

The main thing to remember is that in each column you want to have the bitrate form a nice progression. Now, this playlist has gotten a little too big to show on one slide, so I'm going to split it over three slides. This one shows you the H.264 variant. On this slide, we have the HEVC variant. Now, notice everything has a different video playlist that it's pointing to, and here's our Dolby Vision variant, and notice that everything here has had the same audio group. So, again, if we wanted to have multiple audio formats, we would need to replicate the video variants for each audio format. And again this wouldn't increase the number of video playlists we had to have. It would just increase the number of entries that we had in the playlist.

Okay. We're almost done. Our last bit is about subtitles and closed captions. Now, you can probably guess how this works.

Our variants need to point at the subtitle and closed caption groups that we're using. So we need to add an attribute to our video variant, and we need to describe the renditions. Now, notice that the closed caption rendition does not have a URI attribute.

That tells the system that the closed caption data is found within the video content, not in a separate playlist. So, there you go, the right thing to do is to give us everything you've got. See, now you understand better what I meant.

Okay. I'd like to quickly summarize the talk we've given today.

Emil talked about key performance indicators, about how to get or compute the values and what they mean. And Zhenheng talked about ways to reduce startup time and how to go about resolving stalls and other errors. And I've talked about how to do master playlists. I'd like to briefly mention the HLS validation tools. They do identify many issues with master and media playlists, and it's worth your time to use them.

As always, you can get more information from the WWDC app or the developer website. That's all we have today. Thanks very much for your attention and time.

[ Applause ]