High Efficiency Image File Format

Hi and welcome to session 513. In this talk you will learn about the lower level details of the new High Efficiency Image File format or HEIF and the many advantages that this new file format standard affords. My name is Davide Concion and I manage the Image Compression Team at Apple. During the talk, we will briefly touch upon the current de facto standard for image compression, a standard that everybody's familiar with JPEG. We will go through the requirements that Apple identified as mandatory for a new image format.

We will explain why we think HEIF is the answer to those requirements and we will get to know some of the flexible tools that HEIF implements. We will then present the reasons why Apple thinks HEVC is the right codec to be used within the HEIF file format.

Let's start with JPEG.

JPEG is still the most popular compression technology for images only present on the web and on consumer electronics devices, such as the DSLR cameras, point-and-shoot cameras and cell phones. Cloud services also use JPEG because of its universal support.

JPEG though has several limitations, among those are the compression efficiency. Several new compression algorithms have been developed in recent years that can shrink the file size much more than JPEG and still maintaining the same objective and subjective quality.

Auxiliary images like alpha or depth are not easily supported.

Also, in recent years new ways to present and display animated images have been developed. Apple Live Photo is one of them. JPEG unfortunately does not support animation. Let's look at history map of compression standards developed by JPEG and ITU/MPEG.

JPEG is really starting to show in its years, especially in terms of compression efficiency when compared to recent advancements. As you can see in the slide, JPEG has been finalized as a standard in 1992, a quarter of a century ago.

Since then, several new compression standards have been developed. The latest one in the least is HEVC. And here is HEIF for comparison in the timeline, which has been finalized in 2015. Apple invested a lot of time to find a successor for JPEG and many options were evaluated. The requirements were extensive. The new format needs to support all the features available in JPEG, but at the same time provide better performance. It needs to be friendly to professional photography tools, the web and the cloud. The new format also needs to be flexible and extensible to cope with the ever-changing photography ecosystem. Here is a list of features Apple considered paramount. The compression needs to be state-of-the-art both on the front. It needs to be competitive with natural images, but also when compressing text or graphics. The format needs to be friendly to hardware accelerated and code and decode operations on modern CPUs, GPUs and ESPs. Performance and power is very high in the list of requirements. It needs to support high depth and wide color gamut which is the new frontier for images captured on consumer devices.

It needs to be able to compress 4:4:4 color sampling and also describe HDR content, including HDR metadata, transfer function and color space definitions.

Auxiliary images for example alpha or depth need to have a commonly defined place in image files. New editing tools will be able to utilize auxiliary data for new presentation and editing experiences. In recent years, the new ways to present and display animated images have been developed. Apple iPhoto is one example. Apple iPhoto includes animated content together with static images.

The new common format needs to store animated information efficiently, ideally using Tempra compression techniques and be able to instruct players about the presentation intention for example, a looping sequence.

The new format needs also to support multiple images in the same file. For example, multi-exposure stacks or stereo images. This is to aid the development and implementation of new computational photography algorithms.

Multiple representations of the same image matching the same file are of great importance. For example, multi-resolution, including progressively increasing level of details or the ability to represent the same image encoded with different codecs.

Tiles are an important tool the new format must implement.

It allows for scalable operation on image of any size. We'll be looking into tiles later in the talk.

The new format needs support for rich metadata associated to each image in the file. And also support for time meta for example, a sequence of images.

There is also desire for a new format to be able to include other metadata types for example, audio or text. Last but not least, the new format should be flexible and extensible enough to provide a solid foundation for the future. We believe that HEIF is the answer to all these requirements.

What is HEIF? HEIF stands for High Efficiency Image File Format.

Version one of the spec became an ISO standard in June of 2015.

Version two should be released imminently. A C reference model for HEIF is available upon request at this link. The reference model is meant to provide guidance for HEIF implementation and to understand the specifications. As a side note, the open source project GPAC MP4Box has recently added basic functionality to part C files. The vodaworld long time ago learned that containers and codes are different entities and there are several advantages in keeping them separated. But historically in the image world containers and codecs are tied together and JPEG is no exception. It makes sense to make the distinction in the image world to get the flexibility to . HEIF does exactly that, it specifies a structural format, a container for individual images, as well as image sequences.

It is built on top of the widely used ISO based media file format which is based on Apple QuickTime technologies.

It also uses and enhances structures defined in the MP4 specification and MPEG-21 specifications. Sequences for example, bursts or animations are stored as tracks or time media MP4 style.

Images coded or derived are stored as items MPEG-21 style. Any compression codec can be included in a HEIF container. The HEIF specification explicitly mentions HEVC, H264 and JPEG in terms of file extensions, types and decoder configuration.

The basic building block of a HEIF file like the ISO based media file format is a data structure called box. A box is comprised of a four-character type, for instance in the example on the right the ftyp box or the metabox or the mdat box.

The size of the box in terms of bytes and the payload of the box. The metabox gives a full description of what is included in the file. The handler type of the metabox for whomever is familiar with ISOBMFF specification is of type PICT indicating to a reader that this metabox handles images. Before going into the anatomy of a HEIF file a note on file extensions.

The standard defines explicitly the file extension of a HEIF file depending on the particular codec being used to compress single images or sequences. The list of extensions can be found in the table above. iOS 11 can capture and store HEIF images using the HEVC codec. Therefore, the extension you will be encountering is .HEIC.

In iOS 11 and macOS 10.13 we support all three single image HEIF flavors for decoding and displaying.

Note also that a HEIF file that includes sequences will have a different extension than a HEIF that contains only single images. We will now dive into the HEIF format and its anatomy. Let's start with the concept of item.

Every element in a HEIF file is an item.

There can be coded items for instance, HEVC encoded frame or tiles.

There can be derived items for instance, an image overlay or an image grid.

There can be metadata items for instance, EXIF, XMP or MPEG-7 metadata. Each item can also come with several properties associated to it. Everything is then connected via structures that link certain items to other items or properties.

Images are items and because multiple images can be stored in the same file the HEIF standard differentiates between them by assigning certain roles. Some of the roles specified in HEIF are listed in the table above. The primary recovery image is the representative image of a file.

The primary image should be displayed when no other information is possible or decodable by a player. Only one primary image can be present in a HEIF file. Other full-resolution images in HEIF files are called master images.

The thumbnail is a small resolution representation of a master image. Multiple thumbnails can be stored in a HEIF for example, with different sizes.

It's a very useful feature for progressive decoding and displaying very high-resolution images. The auxiliary image is an image that complements a master image. For example, an alpha plane or a depth map. Auxiliary images can assist in displaying master images, but are not typically displayed. A hidden image is an image that should never be displayed. It can be present in the file for example, as an input image of a derived image. iOS 11 HEIF implementation uses extensively hidden images which are called tiles. Each tile is used to compose the final master or canvas image. Now derived manager is an image that is rendered by an indicated operation being performed on other input images. For instance, the canvas image described before is rendered after stitching together multiple tiles.

Equivalent images are alternative images for instance, encoded with a different codec. A server could distribute the same input content to players that may have different decoding capabilities. Once the role has been defined for each image properties can be associated to them.

Properties are either descriptive or transformative. They can also be essential for example, the codec initialization info or nonessential. The table above provides a non-exhaustive list of descriptive properties for images inside a HEIF file. All the usual suspect information can be found in there like the image size, the color information, the type of auxiliary image which can be alpha or depth and also the configuration parameters to initialize the decoder. The table above provides a non-exhaustive list of transformative properties. The presence of these properties instructs a HEIF that the image needs to go through extra steps before being displayed. For example, the clean aperture property instructs a HEIF reader that the crop operation must be performed before rending the final image. All the properties for each image are grouped together in the same item property box.

Each image can then be associated to which property via the association box. We will use an example to describe how the association works.

The above HEIF container on the left describes the file with one main image and one thumbnail. The main image is comprised of four tiles.

The item property box or ipco box contains all the decoder configuration and the sizes of the main image, the tiles and the thumbnails. Note that the order matters for this box. The association box or ipma box on the right groups properties nicely based on their position with the item ID in the file. As explained before, there is a total of six items in the file, one image, four tiles and one thumbnail.

Items 1 through 4 are the tiles, these are hidden images with properties in position one, the decoder configuration and position two, the size which is 500 by 500 pixel.

Item five is the main image, only the size property is defined since this is this is a derived image. The size is 1,000 by 1,000 pixels.

Next, we will briefly talk about image sequences in HEIF. When sequences are embedded in a HEIF file the move box and it's sub boxes are also present in the file.

The move boxes fully described in the ISO MP4 file format specification from which HEIF derives.

Each sequence of images or samples is described via the trak box where all the timing information to play back the track is included.

HEIF specifies a new track handler for picture called pict.

The key difference is that while the timing information given for a video or an audio track is used to synchronize the playback the timing information an image sequence track can represent either the capture time for example, a burst or the suggested display time for example, to derive a slideshow.

Roles can be used for image sequences to. For example, a HEIF file could embed a track of thumbnails or a track of auxiliary images associated with the master track.

One of the most important HEIF features is the ability to control the playback by signaling in the file the intent of the creator. For example, an edit list enables modifying the playback order and pace of each sample.

HEIF also allows indicating edit list repetitions for example, for looping animations.

The repetition can be indicated to last for a certain duration or be infinite. Given that the ISO tracks can be used in HEIF files interframe prediction is also available. Inter prediction is the ability to remove coded information by predicting the content of the current frame from similar frames in the past or in the future. This gives a tremendous advantage in terms of compression.

Inter prediction can also introduce a delay decode time because the previous frames must be decoded first before being able to decode the current frame.

HEIF allows inter prediction, but also includes constraints in the file to limit frame interdependencies. For instance, each predicted image can be restricted to point only to unknown predicted image or inter. In this case, the time to decode each frame in a sequence becomes deterministic. Last but not least, a HEIF image can be subdivided into tiles. Tiles are rectangular regions within an image. They are completely independent items in a HEIF file and they can be of different or same size. If their size is different a relative location property describes their position in the final image. If their size is the same the final image is described as a grid. Several reasons why tiles make HEIF extremely flexible.

A player can exploit parallelism and decode time. For example, each tile can be separately and independently decoded.

Tiles can be used to reduce memory consumption when resizing an image rather than decoding the whole much and then apply a rescale operation each tile can be independently decoded and rescaled and then placed in a smaller buffer for rendering.

Cropping becomes very fast because a player does not need to decode the whole image to extract a certain region. This property is extremely useful for zooming operation. For instance, a gigapixel image could be decoded and displayed and zoomed in with ease without the need to decode the whole image into a multi gigabit buffer. Of note, the tiles can be used also as an encoding tool. A smart encoder can make different decision based on the content of each tile. Apple HEIF implementation uses tiles extensively. Note though that the HEVC specification also supports subdividing a frame into tiles as a parallelization tool. Apple does not use tiles in HEVC parlance, but rather each tile is a whole HEVC frame, we call them system tiles.

Next, we will talk about HEVC, the codec Apple has chosen to compress HEIF photos. Two of the major reasons for selecting HEVC. First, HEVC is the latest technology in the compression standard world. With HEVC we see an average of 2X compression compared to JPEG containing the same visual quality. Second, HEVC hardware support is becoming available in most CPUs and GPUs. For instance, HEVC hardware support is available from the sixth generation Intel core processors. This means except means exceptional performance without sacrificing battery life.

Several inter coding tools have been added to the standard that allow HEVC to outperform JPEG. In the next few slides we will mention some. You will notice that the common theme here is flexibility. First, the block size.

JPEG divides each image into a grid of blocks of 8 by 8 pixels. These blocks are then described transformed and quantized. HEVC has the flexibility of being able to divide an image in blocks that are 64 by 64 pixels down to 4 by 4 pixels. The transform size is also flexible within the block.

A new optional discrete transform has been added to the standard and three possible scanning orders are available to group coded coefficients. Next, the block prediction.

JPEG allows the top left corner coefficient also called the DC component or the constant component of an 8 by 8 block to be predicted from the block on the left.

HEVC adds the flexibility to predict every pixel value within a block. Up to 35 angular predictions are available. Being able to remove redundant information in a block by exploiting similar information available in neighboring blocks is one of the most efficient tools inside HEVC.

Entropy coding.

JPEG uses Huffman coding as the engine for statistical encoding. The idea is to assign variable length codes to input coefficient.

With shorter length codes assigned to coefficients with higher frequency.

HEVC on the other hand, employs an arithmetic coder called CABAC which stands for Context Adaptive Binary Arithmetic Coding. CABAC is notable for providing much better compression than most other entropy encoding algorithms.

Quantization.

Quantization is a C compression technique achieved by compressing a range of values to a single quantum value.

JPEG utilizes global quantization matrixes for each 8 by 8 block.

HEVC on top of the quantization matrix adds the flexibility of assigning a different quantization parameter for each block. This allows smart encoding algorithms to compress more areas of an image while the human visual system is less susceptible to detect artifacts. For instance, high-frequency content. Next is the blocking, a tool that is available only in HEVC.

Blocking artifacts are visible discontinuities occurring at block boundaries. The HEVC deblocking filter is a filter applied to the pixels around the block edges to smoothen the transition and get more pleasing visual results. SAO which stands for Sample Adaptive Offset is an extra filtering step available in HEVC that is applied to the output of the deblocking filter to further improve the quality. It's a local filter that can attenuate bringing artifacts or changes in sample intensity of some areas of a picture for a better visual quality.

Both these techniques allow for more pleasing images, especially when the compression is very high.

We have gone through several HEIF and HEVC features and tools, I wanted to take a second to mention a few characteristics of HEIF files captured on iOS 11.

First, the extension for HEIF images captured with iOS 11 will be .HEIC because of the HEVC codec.

The HEVC profile utilized to compress images is the main still profile. Also, we use HEVC monochrome profile for depth data.

Images are encoded using tiles that are 512 by 512 pixels. They are positioned in a grid fashion to cover the whole image.

The thumbnail is a 320 by 240 image HEVC encoded.

It is four times the size of a common 160 by 120 JPEG thumbnail and this is to help showing better thumbnail quality when images are displayed on modern screens with high pixel density.

EXIF metadata is part of the HEIF file like JPEG for backward compatibility.

Depth data is stored as an auxiliary image and the metadata pertinent to depth is stored as XMP payload associated with the depth image. Last, a note about file creation.

The HEIF standard does not mandate any order for the boxes a reader could find at the top level of a HEIF file, but we found that ordering them in a certain way greatly helps parsers and decoders. For example, having the thumbnail early in the file would allow parsing and display huge amount of HEIF images without the need to parse the whole file.

For transmission or web application once the metabox is received all the information for the file is available and therefore readers can start configuring the decoding and display pipelines before having received the whole coded data.

Let's summarize what we have learned today. The photography world needs a better image file format to replace the rather old JPEG. We looked at the extensive list of requirements that Apple considered paramount when searching for a JPEG replacement. We believe HEIF is the answer for all the requirements. Its flexibility allows to handle with ease and elegance the advancements available in iOS 11 and its extensibility also allows HEIF to be a solid foundation for the future. We then analyzed the various features available in the HEIF standard.

And finally, we looked at the HEVC tools that make it the best choice, both in terms of compression efficiency and friendliness toward hardware architecture for performance and power. For more information, please visit the URL for the High Efficiency Image File format session 513.

And if you're still at the show we invite you to visit the two related sessions about HEIF and HEVC. Thank you for watching the talk and enjoy the rest of WWDC 2017.