Support Apple Pro Display XDR in your apps

Hello, everyone, and thanks for joining us today. My name is Pranav, and I work on the media quality team here at Apple. I'm very excited today to talk about the Apple Pro Display XDR. In this talk, we are going to guide app developers as well as professional users on how to work with this amazing display and use it in the most recommended way.

The Pro Display XDR is a high-grade reference monitor. It's a 32-inch LCD display, has over 20 million pixels, and with a 6K resolution, this is the largest retina display Apple has ever made. It has a P3 wide color gamut and supports true 10-bit color. The display can get as bright as 1600 nits, and it can even maintain 1000 nits of full-screen luminance indefinitely. This makes it an ideal display for all your HDR workflows. We have also built reference modes into the display, and all of these modes are calibrated at the factory. For you, this means you can now work on a display which has a high color accuracy and comes packed with features for all your postproduction needs.

We have built every element on this display for pros. It is designed to support professional workflows such as video editing, photography, color grading, visual effects, 3D animation and game development.

Now, a few of the pro apps out there have already been customized to work with this display. And here you can see a few of them. As an example, Blackmagic Design has enabled the support of EDR in Resolve. This has enabled Resolve to display both extended range and SDR content simultaneously. They have also added the ability to assign unique color spaces to the input footage, the timeline working space and the output. And while Adobe Premiere Pro and Foundry Nuke have already implemented preliminary support for EDR, full EDR support for these apps will be coming soon.

In this session, we want to help other pro app developers create and customize their own app, making full use of this display's capabilities.

At the same time, we want to provide our pro users with the best practices on using this display to ensure their workflows are set up correctly.

With that aim in mind, here's what we'll cover today.

We're going to start by understanding what reference modes are and learn about the different characteristics of reference modes available on this display.

We'll then talk about some of the key concepts related to color management and EDR which are crucial when developing pro apps for this display. Next, for our pro users, we will discuss some of the recent features we have shipped and how these features can make their workflows both more capable and accurate.

And finally, we'll show you how you should measure the front-of-screen response of this display and ensure that your content is being rendered correctly as you work in any of the reference modes.

So let's get started.

Well, what are reference modes? For each of the reference modes, we have configured the luminance, color and gamma response of the display itself, so when you switch from one reference mode to another, the display changes its own behavior to adapt to the particular workflow.

And since each of these modes are calibrated, you can expect high color accuracy for all your reference modes.

Now given this, a reference mode is much more than simply plugging in an ICC display profile. In fact, as you switch between reference modes, ICC profiles get automatically populated, which helps the display work in conjunction with color management provided by macOS. And like with any color managed system, your media surfaces need to be tagged with color descriptors. And we'll discuss this in detail soon.

The display comes with several built-in reference modes out of the box. And there are two categories of reference modes.

The first is Apple-designed modes, which reflect the characteristics of our internal Mac displays. So when you are in one of the Apple-designed modes, you will notice dynamic adaptation features, such as True Tone or auto brightness, can be enabled or disabled.

These adaptive technologies are great because they facilitate viewing even in a non-reference viewing environment. However, professional users who are authoring content in a reference environment would want to work in reference modes where these dynamic features are strictly disabled.

For that, we have standards-based modes. These have been designed to support specific postproduction workflows which are in alignment with common industry specifications and assume that you are working in the reference viewing environment.

Viewing condition is extremely important when using these modes. Typically most workflows require you to work in a dim viewing environment. Let's look at a few of the most popular reference modes in detail. This is the HDTV video reference mode built using the ITU BT.709 and BT.1886 recommendations. Now, while our display can do up to 1600 nits and a P3 wide color gamut, you do not need all of it when working in an SDR workflow. So when you switch to this mode, we intentionally restrict the peak luminance of the display to 100 nits and the color volume to a Rec.709 color gamut. Also all dynamic controls are disabled, and you should use this reference mode in a dim viewing environment. Next let's check out the HDR video reference mode. Here we see a P3 wide color gamut and a peak HDR luminance of 1000 nits. And you can have 1000 nits of full-screen luminance indefinitely. For you, this means you can now work on your HDR workflows for as long as you want and not worry about the patch size or a huge drop in luminance over time.

In the pro display XDR reference mode, we make full use of the display's capabilities. Here we have an HDR peak luminance of 1600 nits for up to 39% of screen area. And your SDR peak luminance can be set up to 500 nits using the brightness slider. Also, all dynamic adaptation features can be enabled, and HDR video content is tone mapped. Now, very often you would want both your SDR and HDR assets to be displayed simultaneously on the same screen. For example, you might want to check your e-mail, which is in SDR, and also play a video which might be in HDR. To enable this, we have come up with a unique way of merging both SDR and HDR elements into one. And we'll discuss soon how we do that.

We will now talk about a few important concepts to be aware of when developing apps for this display.

To understand reference modes more, we need to understand the different pixel processing stages in our media pipeline. Let me show you what happens when we are in one of the SDR reference modes.

Our SDR video content is decoded by core media and tagged with the color descriptors in the file. Next, both UI and SDR video get mapped into a common color representation for composition. The composed interface is then mapped to the display's transfer function and color space before sending it out to the pro display.

When we are working in an HDR reference mode, such as the HDR video or the pro display XDR mode, the Mac and the display communicate over a PQ link. Again, SDR content and UI is decoded to linear light using the tagged color space and gamma.

HDR content is decoded to linear light using the PQ or HLG transfer function. All compositing is done in an EDR space. Let's discuss briefly what EDR is.

EDR stands for extended dynamic range, and it's our approach to simultaneously represent SDR and HDR scenes using extended range. EDR is an SDR referred working space, such that SDR content remains unchanged in its logical 0-to-1 range while EDR allows brighter values above 1.0. The range above EDR 1.0, typically referred to as headroom, is generally reserved to represent specular highlights and emissive surfaces in the scene.

Finally, the composited frame is mapped to the display's color space and PQ transfer function and sent to the display.

Now we have seen the different reference modes behave differently and make use of the color management provided by macOS. As a result of this, it is extremely important to tag your media content, both images and videos, correctly.

This will ensure that the media data gets interpreted correctly by macOS, which is a color managed system. For images, you would assign a color profile to it, and for videos, you would tag it with standard color descriptors.

Let's look at the color descriptors in a video file. You can use QuickTime Player's Movie Inspector to read the color tags associated with the video file.

The color primaries tag specifies the mapping from linear RGB to XYZ space.

The transfer function tag specifies the electrooptical transfer function to convert nonlinear R'G'B' code words to linear RGB space. And finally, the Y'CbCr matrix specifies the mapping between Y'CbCr and R'G'B' space. If you are using AVFoundation to tag your content, I recommend watching this year's WWDC session on exporting HDR media.

For reference modes on pro display XDR, here's a table of recommended color descriptors you should be using when tagging your video files. These tags ensure that your content gets interpreted correctly in the reference mode you are working in.

Now, if your app is using AVFoundation framework, it will correctly identify the reference mode and provide the necessary color management required for each of the modes. First, Y'CbCr code word in the source gets decoded by core media and converted to nonlinear R'G'B' code word in source color space.

This is then mapped to the preset's transfer function and color space.

The display can then use the preset's characteristics to linearize the input signal.

If your app hardware uses a custom video pipeline, here are a few things that you need to do. First, you need to convert code words in Y'CbCr space to R'G'B' using the NCLC tag specified for the Y'CbCr matrix.

The R'G'B' code words should then be linearized to niche value and mapped into EDR. This mapping into EDR should be done such that EDR 1.0 is mapped to reference white in the source. Pixels which are brighter than reference white can then be mapped to EDR values greater than 1.0.

The display pipeline then takes care of mapping EDR to preset's color space and feeding that to the display.

Now, based on the reference mode the user is working in, you will want to manage the mapping to EDR differently. This will help you tailor your app's behavior to take advantage of it. To correctly identify which reference mode the user is working in, you can make use of the EDR parameters. Let's understand each of the three headroom parameters. Potential headroom tells you the maximum headroom possible in the current display configuration. And it might vary from one reference mode to another.

For example, the HDR video reference mode has a peak HDR luminance of 1000 nits and a peak SDR luminance of 100 nits. This implies it has a potential headroom of 10.0. The HDTV video mode, which has a peak SDR and HDR luminance of 100 nits, has a potential headroom of 1.0.

The headroom parameter can raise the amount of headroom currently available, and this can be less than or equal to the potential headroom. Note that as brightness is increased in a reference mode, headroom can decrease. As an example, the Pro Display XDR preset has a peak HDR luminance of 1600 nits. If the brightness slider is set to an SDR peak of 200 nits, the headroom is 8.0, while the potential headroom is still 16. Finally, reference headroom is the amount of headroom in which you can expect accurate reproduction of requested luminance. Reference headroom can be less than or equal to the headroom value.

For example, in the HDR video and the HDTV video reference modes, we guarantee accurate reproduction for up to 1000 nits and 100 nits, respectively. Hence the reference headroom is the same as headroom for these modes. For Apple-designed modes, the reference headroom is zero.

And here's how you can do it in code. First, we create an NSScreen object which points to the Pro Display XDR screen. Then, we can query the potentialHeadroom, headroom and referenceHeadroom parameters.

Now that your video pipeline is set up, you would want to evaluate your app's color performance for correctness. One way to do that is to measure the front-of-screen response of the display using a measurement instrument such as a spectroradiometer. Here's a flowchart summarizing the different stages involved.

As we discussed previously, color management is performed in macOS to map the input signal to the preset's color space. The Pro Display XDR then takes in the R'G'B' code words as input and generates the light spectrum.

A spectroradiometer can then be used to measure the light output to derive luminance and chromaticity values.

To help with this process, we have provided video test patterns which are available on the AVFoundation developer web page. These test patterns are QuickTime movie files tagged with the correct NCLC and static metadata for HDR 10 files. And this includes a 21-step luminance ramp, 100% amplitude color primaries and color secondaries.

We have also included with it the reference luminance and chromaticity values that you should expect when these test patterns are displayed in the reference mode they're designed for.

We will now talk about deploying Pro Display XDR and show how our pro users can make use of some of the new features we have released to make their workflows both more capable and more accurate.

So Pro Display XDR is calibrated at the factory and is ready to be used out of the box. We also have a list of built-in reference modes which have been designed for the most popular workflows.

Together with these, we now also support an array of customization options.

Let's start by talking about customized reference modes.

Now, although the display comes with a list of reference modes built into it, but what if your unique workflow, it needs a different combination of transfer function, white point and color gamut? Or if you want to use a different maximum luminance target? For that, we have customized reference modes available in macOS 10.15.4 and later. With custom reference modes, you can now customize these parameters to create a unique reference mode.

Let me show you how we can create a custom reference mode. Let's imagine you need to create a workflow with 600 nits of maximum HDR luminance. Also let's say you would like your UI to be brighter than 100 nits. Maybe 200 nits. You should first select Displays from System Preferences. On the window title Pro Display XDR, select Customize Presets from the presets list.

Now choose a base preset from which you want to create your custom preset.

Next, click on the plus icon.

Here you can name your preset and edit the description.

From the available options, you can choose the color gamut, the white point and the SDR transfer function.

You can also explicitly apply a system gamma boost.

Finally, we change the maximum HDR luminance to 600 nits and the maximum SDR luminance to 200 nits and save the preset. We have now created our unique custom reference mode. Let's look at the next feature.

Now, very often measurement instruments have slight differences from one unit to another. And if all your workflows have been set up to work with your in-house instrument, you may want to fine-tune your Pro Display XDR to match with other displays. For those of you who have access to high-end calibrated spectroradiometers, you can now do this using the fine-tune calibration feature. Fine-tune calibration is available in macOS 10.15.6 and later. This feature allows you to adjust your display's luminance and white point so that it better aligns with your in-house spectroradiometer and matches other monitors. Let's see how we can use this on Pro Display XDR.

First, we go to System Preferences and choose Displays. In the preset list pop-up menu, we can see the Fine-Tune Calibration option available.

Here, we need to enter our measured and target luminance and white point values. In the next section, we will discuss in detail how we can obtain these.

Now we know that as content creators, you want to make sure that your display is behaving the way you expect it to. So in this section, we will discuss how you can objectively evaluate that your content is being rendered correctly.

Here is a flowchart describing our recommended process for front-of-screen evaluation when using the Pro Display XDR.

First, we need to create our test patterns and calculate the reference values we expect when the test patterns are displayed on a reference monitor and measured using a calibrated spectroradiometer.

Then, we set up our measuring instrument and the environment.

For accurate results, we recommend you use high-end calibrated spectroradiometers while making measurements. Here is a list of a few spectroradiometers that we recommend using.

Also, make sure your instrument is configured correctly. For example, if you're using a Photo Research PR-740 spectroradiometer, you might want to set your instrument's bandwidth to four nanometers and set the average number of readings for each measurement to be three or higher.

Your low luminance measurements might get affected if stray light interferes with your measurement. For best results, make sure your environment is completely dark.

And to avoid any thermal radiation, we recommend working in a room with controlled temperature, typically less than 25 degrees Celsius.

The next step is to display the test patterns on screen and make measurements.

Once you have the measurements and reference values, you should perform an error analysis. Very high errors might indicate an error in the process. This can be either your test patterns aren't tagged correctly, your reference mode selection is incorrect or your instrument and environment isn't set up properly.

For error analysis, you should compute perceptual color difference metrics to know how visible the differences are between the measured and reference values.

In this example, I'm using a color difference calculator we have built which computes several perceptual color difference metrics. We will be providing you with the sample code for this.

The calculator takes in reference and measured values and computes delta-E ITP, delta-E 2000 and delta PQ metrics.

You can define your input signals from a list of color spaces, color gamuts...

transfer functions and representations.

Since our input signals are spectroradiometer measurements, we will select the Yxy color space.

Next, we enter the reference luminance and chromaticity values. And for this example, we have chosen 94 nits, D65 as our target.

We then enter the measured luminance and chromaticity values, which are slightly off the target, as you can see. Finally, we compute the error metrics. We can repeat this process for the entire 21-step luminance ramp to compute error metrics for each measurement point.

This is an example of a chart showing delta-E ITP values for the entire ramp. And you can see that the maximum error in this case occurs for a target luminance of around 94 nits.

Next, if the errors are perceptually insignificant, we have validated our workflow. But if the errors in luminance and white point are perceptually significant, we can improve those using the fine-tune calibration feature.

We will now use the maximum error case to fine-tune our display.

First we enter the measured white point and luminance values.

Then we enter the target white point and luminance values.

Finally, we can add a description and hit "okay" to fine-tune the calibration of our display.

We have fine-tuned the calibration of our display now. For a final validation, we'll make measurements and perform an error analysis again.

This is an example to show you the improvement in delta-E ITP that you might see after fine-tune calibration. The entire luminance ramp will match your in-house spectroradiometer much more closely now.

We will soon be adding support for performing an in-field recalibration of your display.

With this feature, you will be able to recalibrate your display for better alignment with your instrument. The calibration process will adjust your display's color primaries, luminance and gamma and will require a high-end spectroradiometer and a dark environment.

And here's the process flowchart updated with in-field recalibration added to the flow.

Here are a few important things that we want you to take away from this talk.

Reference modes on Pro Display XDR are much more than using an ICC display profile and provides a way to configure the luminance, color and gamma response of the display itself. Using color management on Mac, each reference mode supports multiple media formats.

You can either use the AVFoundation framework or color manage your own video pipeline. With customized reference modes, now you can create your own reference mode and work on your unique workflows. You can evaluate your workflow using Apple-provided test patterns and reference values. And finally, you can use the in-field recalibration and fine-tuning features to adjust your display's color primaries, luminance and white point for better alignment with your in-house measurement instrument. Thank you.