HEIF：Technical information

https://nokiatech.github.io/heif/technical.html

Technical information

Introduction

High Efficiency Image File Format (HEIF, ISO/IEC 23008-12) specifies the storage of individual images, image sequences and their metadata into a container file conforming to the ISO Base Media File Format (ISO/IEC 14496-12). HEIF includes the storage specification of HEVC intra images and HEVC image sequences in which inter prediction is applied in a constrained manner. Use cases supported by HEIF include:

• Storage of still images as well as collection of images in a single container file,
• Storage of burst photos,
• Storage and efficient representation of video animations and cinemagraphs.
• Support for simultaneous capture of video and still images, i.e. storing still images and timed image sequences into the same file.
• Storage of images derived from other images of the file, either as derived images represented by non-destructive image editing operations or as pre-computed derived images.
• Storage of focal and exposure stacks into the same container file.
• Support for storing other media, such as audio and timed text, into the same container file with timed image sequences and synchronizing their playback.

As HEVC provides support for various chroma formats and sample fidelities up to lossless coding, the format can serve the whole spectrum of use cases from today's consumer devices storing images typically at 8 bits per sample to high-end professional devices with sample fidelity and dynamic range requirements going all the way up to 16 bits per sample.

Computational photography forms a new category of use cases that can benefit from the HEIF file format. Now a set of related images can be stored in a single file with associated metadata indicating relationships between different pictures.

HEIF specifies a structural format, from which codec-specific image formats can be derived. HEIF also includes the specification for encapsulating images and image sequences conforming to the High Efficiency Video Coding (HEVC, ISO/IEC 23008-2 | ITU-T Rec. H.265).

In ISOBMFF, a continuous or timed media or metadata stream forms a track, whereas static media or metadata is stored as items. Consequently, HEIF has the following basic design:

• 1. Still images are stored as items. All image items are independently coded and do not depend on any other item in their decoding. Any number of image items can be included in the same file.
• 2. Image sequences are stored as tracks. An image sequence track can be indicated to be displayed either as a timed sequence or in a non-timed manner, such as a gallery of images. An image sequence track may be used instead of image items when there is coding dependency between images.

A file may contain both image items and image sequence tracks along with other media. For example, it is possible to create a file, which includes image items or image sequence tracks conforming to HEIF, and audio and timed text tracks conforming to any derivative format of the ISOBMFF.

Image I. ISOBMFF compliant file high level structure with meta box and items

Files conforming to ISOBMFF consist of a sequence of data structures called boxes, each containing a four-character code (4CC) indicating the type of the box, the size of the box in terms of bytes, and the payload of the box. Boxes may be nested, i.e. a box may contain other boxes. ISOBMFF and HEIF specify constraints on the allowed box order and hierarchy.

HEIF Related Brands and MIME Types

Files conforming to HEIF start with a FileTypeBox as defined in the ISOBMFF standard, which contains a list of brands the file complies with. As the FileTypeBox is located at the start of the file, it provides easily accessible indications of the file contents to media players. Each brand is identified by its unique four-character code. The specification of a brand can include requirements and constraints for files of the brand and for file players supporting the brand. A brand included in the FileTypeBox permits a player that supports the requirements of the brand to play the file.

The brands specified in the HEIF standard are presented in Table I. The HEIF standard specifies the 'mif1' and 'msf1' structural brands. Additionally, HEVC-specific brands are specified as listed in Table I. The dedicated brand names, 'heic' and 'hevc' indicates that HEVC Main Profile is utilized.

Table I. Brands, MIME subtypes, and file extensions for HEIF.

BrandCoding formatImage or sequence?MIME TypeMIME subtypeFile extensionmif1Anyimageimageheif.heifmsf1Anysequenceimageheif-sequence.heifheicHEVC (Main or Main Still Picture profile)imageimageheic.heicheixHEVC (Main 10 or format range extensions profile)imageimageheic.heichevcHEVC (Main or Main Still Picture profile)sequenceimageheic-sequence.heichevxHEVC (Main 10 or format range extensions profile)sequenceimageheic-sequence.heic

Internet media types, also known as MIME (Multipurpose Internet Mail Extensions) types, are used by various applications to identify the type of a resource or a file. MIME types consist of a media type ('image' in the case of HEIF files), a subtype, and zero or more optional parameters. For multi-purpose files, the selection of the subtype can be made on the basis of the primary use of the file.

An optional codecs MIME parameter can be present to indicate the used coding formats of the tracks and items present in the file. The codecs MIME parameter also includes the profile-tier-level value to which an HEVC-coded image item or an image sequence track conforms. More information about the MIME type registration and optional parameter can be found in Annex D and Annex E of the HEIF standard.

Comparison to other Image Formats

Table IX provides a comparison of the features of HEIF to other selected image formats. It can be observed that HEIF is more extensible and comprehensive than the other compared file formats. Particularly the possibility to include other media types, the advanced multi-picture features, and the support for non-destructive editing make HEIF more advanced than the other formats. The rich set of features make HEIF suitable for a broad range of devices and applications, including for example burst photography.

The references used to conclude the information in Table II are included in Table X.

Table II. Comparison of the features of some image file formats.

 .heicJPEG/ExifPNGGIF (89a)WebPJPEG-XR / TIFFJPEG-XR / JPXBPGFormats and extensibility        Base container file formatISOBMFFTIFF--RIFFTIFF- ⁴-Lossy compressionYes (HEVC)Yes (JPEG)NoNoYes (VP8)YesYesYes (HEVC¹⁰)Lossless compressionYes (HEVC)Yes (TIFF Rev 6.0)Yes (PNG)¹Yes (GIF)¹Yes (VP8L)YesYesYes (HEVC¹⁰)Extensible to other coding formatsYesYes⁸NoNoNoYes⁸Yes⁵NoMetadata format (on top of internal)Exif, XMP, MPEG-7Exif--Exif, XMPExif, XMPJPX, (XMP)⁶Exif, XMPExtensible to other metadata formatsYesNoNoNoNoNoYes (XML-based)YesOther media types (audio, text, etc.)YesAudio²NoNoNoNoYes⁷NoMulti-picture features        Multiple images in the same fileYesNo¹¹NoYes³Yes³NoYesYes⁹Image sequences / animationsYesNoNoYesYesNoYesYesImage codingYesNoNoNoNoNoNoYesDerived images        Multiple-of-90-degree rotationsYesYesNoNoNoYesYesNoCroppingYesNoNoNoNoNoYesNoTiling/overlayingYesNoNoNoYesNoYesNoExtensible to other editing operationsYesNoNoNoNoNoNoNoAuxiliary picture information        Transparency (alpha plane)YesNoYesNo¹²YesYesYesYesThumbnail imageYesYesNoNoNoYesYesYes

¹ In GIF and indexed color PNG encoding, lossy color quantization is applied while the color-quantized image is losslessly compressed.
² PCM, µ-Law PCM and ADPCM encapsulated in RIFF WAV
³ Only for animations and tiling/overlaying
⁴ JPX is a box-structured format compatible with ISOBMFF. However, only the File Type box is common in JPX and ISOBMFF.
⁵ Encapsulation of JPEG-2000 and JPEG-XR have been specified for JPX container. Mappings for other codecs could be similarly specified.
⁶ JPX (ITU-T T.800 and T.801) specifies an own metadata schema, but is capable of carrying an XML formatted metadata, such as XMP.
⁷ JPX can contain media complying with ISOBMFF (or derivatives thereof). No accurate synchronization between JPX animations and other media.
⁸ TIFF as a container format facilitates extensions to other coding formats.
⁹ Only for animations, thumbnails, and alpha planes. Non-timed image collections not supported.
¹⁰ HEVC Main 4:4:4 16 Still Picture profile, Level 8.5, with additional constraints
¹¹ Can be enabled through the MP extension
¹² A palette index for fully transparency can be specified

It is acknowledged that a summary such as that in Table II might be somewhat incomplete when it comes to features of different formats. For example, the table does not cover some of the extensions of JPEG. We welcome feedback and corrections to the table.

Table III. References for the compared image file formats

Image formatVersion or dateReference and/or URLHEIF (.heic)03/2015ISO/IEC 23008-12JPEG ISO/IEC 10918-1 | ITU-T Rec. T.81
http://www.w3.org/Graphics/JPEG/itu-t81.pdfExif http://www.cipa.jp/std/documents/e/DC-008-2012_E.pdfPNG http://www.w3.org/TR/PNG/GIF89ahttp://www.w3.org/Graphics/GIF/spec-gif89a.txtWebP https://developers.google.com/speed/webp/docs/riff_container
VP8L:
https://developers.google.com/speed/webp/docs/webp_lossless_bitstream_specificationJPEG-XR ISO/IEC 29199-2 | ITU-T Rec. T.832
ISO/IEC 15444-2 | ITU-T Rec. T.801 (for JPX)BPG0.9.5http://bellard.org/bpg/bpg_spec.txt

The file size of container files are directly affected by the compression performance of the image/video codec being utilized. Such a comparison can be found in Annex B of this document.

Compression Performance

Table IV illustrates coding efficiency of HEVC intra coding with respect to well-known still picture codecs. The results indicate that JPEG would require on average 139 % higher bitrate than HEVC (i.e. 2.39 times the file size) in order to achieve the same objective picture quality. For JPEG-XR and JPEG-2000 the average increase in bitrates are 66 % and 44 %, respectively.

Table IV. HEVC intra coding performance with respect to legacy formats. Bitrate increase required to achieve the objective quality provided by HEVC intra coding is reported for each test category.

ClassResolutionCharasteristicsJPEGJPEG XRJPEG 2000Class A2560x1600Cropped 4Kx2K sequences for Ultra HDTV services87 %44 %48 %Class B1920x1080High resolution sequences for streaming and broadcast services124 %62 %15 %Class C832x480Medium resolution sequences for Internet/mobile video services122 %53 %50 %Class C832x480Medium resolution sequences for Internet/mobile video services122 %53 %50 %Class D416x240Low resolution sequences for services to resource constrained devices110 %47 %43 %Class E1280x720720p sequences for video conferencing applications170 %73 %23 %Class F1024x768, 1280x720Computer screen content and computer generated content223 %118 %87 %Average  139 %66 %44 %

Table V indicate that one can expect that for natural content the restricted inter coding can typically provide two to three times better compression than intra picture coding. In special cases like animations where majority of the scene is static the compression efficiency can significantly exceed those levels and be tens of times more efficient than intra coding.

Table V. Coding efficiency improvements provided by low latency predictive coding of the HEVC Image File Format. Bitrate impact and coding gain are reported with respect to HEVC intra coding.

ContentTypeFramesBitrate changeCoding gainClass AImage burst8-46 %1.9Class BImage burst8-51 %2.0Class CImage burst8-60 %2.5Class DImage burst8-63 %2.7Class EImage burst8-79 %4.8Class FImage burst8-55 %2.2MemorialExposure stack16-29 %1.4MersuFocal stack13-25 %1.3Car&TractorCinemagraph48-97 %33.3

Images in HEIF Files

Multiple images can be stored in a HEIF file. It can be useful to differentiate between them by assigning them certain roles. The roles specified in HEIF are listed and described inTable VI. Note that a single image can be associated with more than one role.

Table VI. Roles of images.

RoleDescriptioncover imageA representative image of the image items and image sequence tracks of the file.
The cover image should be displayed when no other information is available on the preference to display the image items of the file. The file can have only one cover image.thumbnail imageA smaller-resolution representation of a master image.auxiliary imageAn image that complements a master image. For example, an alpha plane or a depth map.
Can assist in displaying the master image but is not typically displayed as such.master imageAn image that is not a thumbnail image or an auxiliary image. Typically represents a full-resolution displayable image.hidden imageAn image that should never be displayed. Can be present in the file for example as an input image for a derived image.pre-derived coded imageA coded image that has been derived from other images.
For example, a high dynamic range image derived from an exposure-bracketed set of images.coded imageA coded representation of an image.derived imageAn image that is represented in a file by an indicated operation to indicated input images and can be obtained by performing the indicated operation to the indicated input images.

Properties of Images

HEIF allows the storage of image properties which are shared among different image items in a compact way. These properties are stored in ItemPropertyContainerBox. There are mainly two types of properties: descriptive and transformative. Descriptive properties provide information about the image item without applying modifications on the image itself. Transformative properties provide information about the transformative modification that needs to be done on the image item. The order of application of these properties to the image items are defined in the standard. Table VII lists currently defined properties. In addition to descriptive image properties, image items can optionally be characterized with metadata items, the format of which follows Exif, XMP, or MPEG-7 metadata.

Table VII. Image Properties

NameTypeDescriptionDecoder configuration and initializationDescriptive PropertyThe information needed to initialize the decoder. The structure of this information is usually defined in the related image coding format specification.Image spatial Extents (‘ispe’)Descriptive Propertyindicates the width and height of the associated image itemPixel Aspect Ratio (‘pasp’)Descriptive Propertyhas the same syntax as the PixelAspectRatioBox as defined in ISO/IEC 14496-12.Color Information (‘colr’)Descriptive Propertyhas the same syntax as the ColourInformationBox as defined in ISO/IEC 14496-12.Pixel Information (‘pixi’)Descriptive Propertyindicates the number and bit depth of colour components in the reconstructed image of the associated image item.Relative Location (‘rloc’)Descriptive Propertyindicates the horizontal and vertical offset of the reconstructed image item relative to the associated image item.Image Properties for Auxiliary Images (‘auxC’)Descriptive PropertyAuxiliary images must be associated with an AuxiliaryTypeProperty.Image Rotation (‘irot’)TransformativeRotation by 90, 180, or 270 degrees.Clean Aperture (‘clap’)TransformativeCropping according to a given cropping rectangle.

Derived Images

Derived images enable non-destructive image editing, where the original coded images are kept in the file, while new images, called derived images, can be introduced by specifying a transformation operation that is applied to one or more input images. HEIF specifies the generic structures used for storing derived images as items as well as a few specific types of derived images. Derived images can also have descriptive or transformative image properties. Item references of type 'dimg' specify the input image(s) of the derived image. The input images can be coded images or derived images. The derived image types specified in the HEIF standard are listed inTable VIII. Other types may be specified in other documents or later versions of the HEIF standard.

Table VIII. Derived Images

NameDescriptionIdentity transformation (‘iden’)Cropping and/or rotation by 90, 180, or 270 degrees, imposed through the respective transformative properties.Image Overlay (‘iovl’)Overlaying any number of input images in indicated order and locations onto the canvas of the output image.Image Grid (‘grid’)Reconstructing a grid of input images of the same width and height.

Here are some examples of Derived Images

Image Metadata

HEIF files allow the storage of the metadata which can be related to images and image sequences. Such metadata can be information related to integrity checks, EXIF or XMP data or MPEG-7 related metadata. For image items, such metadata can be stored as a metadata item and reference the related image items with a ‘cdsc’ item reference. For image sequences, timed metadata tracks can refer to an image sequence track with a ‘cdsc’ track reference.

Indicating Alternative Representations

HEIF files can contain several image items representing the same image content (i.e. alternative representations). A media player may select one of the alternative representations of the same image content for displaying. A mechanism, known as entity grouping (defined in the GroupsListBox), is used for indicating alternate groups, which can contain both image items and media tracks (image sequences). The capability to define alternate groups between tracks and image items provides a unique mechanism for display initialization based on preferences of the media player and the content creator. Examples where alternate groups can be useful are:

• The same original image is coded into two or more image items with different properties, such as spatial resolution, bit-depth, or color gamut.
• The same original image is coded into two or more image items with a different coding format or profile.
• A timed image sequence, such as a cinemagraph, is an alternative to an image item.

Image sequences

Images that have a particular correlation (e.g. image bursts or animations like cinemagraphs) can be efficiently stored in HEIF files, thanks to the inherited media track features of ISOBMFF. Such images are called image sequences and they can reside in the same file with image items. Table IX lists the roles of the image sequence tracks which are currently defined.

Table IX. Roles of image sequence tracks.

RoleDescriptionthumbnail image sequenceA smaller-resolution representation of a master image sequence. Tauxiliary image sequenceAn image sequence that complements a master image sequence. For example, a sequence of alpha plane or depth map images. Can assist in displaying the master image sequence but is not typically displayed as such.master image sequenceAn image sequence that is not a thumbnail image sequence or an auxiliary image sequence. Typically contains full-resolution displayable images.

HEIF enables the usage of Inter-picture prediction for compact storage of image sequences. Moreover, image sequence use cases may require faster access to individual images and the ability to edit individual images without affecting any other images. HEIF therefore includes the following two features:

• CodingConstraintsBox to indicate prediction constraints which applies to all the linked samples. For example, it can be indicated that all reference images used for inter-picture prediction are intra-coded images.
• DirectReferenceSamplesList to have sample-wise listing of referenced samples using the sample grouping mechanism of ISOBMFF. This feature enables decoding of only those samples that are needed for accessing a particular sample (e.g. random access) and avoids the need of sequential decoding of each sample starting from the previous intra-coded image.

Processing of image items by file players

Figure 1 illustrates how a file player processes the coded images and the derived images included in a file. The file player decodes a coded image into a reconstructed image. Similarly, the file player applies the operation of the derived image to the indicated one or more input images to obtain the respective reconstructed image. The descriptive image properties generally describe the reconstructed image, with the exception of the decoder configuration and initialization information, which is associated with the coded image. The transformative image properties, if any, are applied to the reconstructed image to obtain an output image. The output image can be displayed, when the coded image or the derived image is not a hidden image. The output image can also act as an input image to derived images.

Figure 1. Operations performed by file players

HEIF File Playback Options

The most important features that enable controlling the playback of an HEIF file are listed inTable X. Some of these features were introduced in the ISOBMFF or ISO/IEC 14496-15 and are explicitly inherited by HEIF, while other features were specifically designed for the HEIF standard.

Table X. Features controlling image sequence playback.

FeatureFirst appeared inDescriptionnon-displayable sampleISO/IEC 14496-15Is never displayed, but can be used as a reference for predicting other images in the track.timed vs. non-timed playbackHEIFIn timed playback, the image sequence is played as video, whereas in non-timed playback the samples of the track are displayed by other means, such as an image gallery. Non-timed playback may be indicated e.g. when a track is used for achieving a better compression efficiency for an exposure stack.edit listISOBMFFA list of ranges of the image sequence track in their playback order. Enables modifying the playback order and pace of samples.loopingHEIFHEIF allows indicating edit list repetition e.g. for looping animations. The repetition can be indicated to last for a certain duration or be infinite.cropping and rotationISOBMFFRectangular cropping and rotation by 90, 180, 270 degrees can be specified.

Storing HEVC Encoded Pictures in HEIF: HEVC Image Files

The HEIF standard includes the specification for encapsulating HEVC-coded images and image sequences into HEIF-compliant files. The specification includes the following aspects:

• The sample entry format and the sample format of HEVC image sequence tracks are identical to those of the HEVC video tracks specified in ISO/IEC 14496-15. For compatibility with HEVC-capable video players, it is therefore advisable to create both an HEVC image sequence track and an HEVC video track that refer to the same coded images. This is called a Dual-Branded File and has an insignificant impact on the file size.
• The format of the decoder configuration information in the HEVC configuration item property is identical to that in the sample entry format. Likewise, the item format is identical to the sample format of HEVC video and image sequence tracks. This enables having the same intra-coded image as an image item and as a sample in an HEVC image sequence or video track.
• The configuration information of alpha planes and depth maps follows the respective supplemental enhancement information of HEVC.