Understanding the Asset Model

Many of AVFoundation’s key features and capabilities relate to playing and processing media assets. The framework models assets using the AVAsset class, which is an abstract, immutable type representing a single media resource. It provides a composite view of a media asset, modeling the static aspects of the media as a whole. An instance of AVAsset can model local file-based media, such as a QuickTime movie or an MP3 audio file, but can also represent an asset progressively downloaded from a remote host or streamed using HTTP Live Streaming (HLS).

AVAsset simplifies working with media in two important ways. First, it provides a level of independence from the media format. It gives you a consistent interface for managing and interacting with your media regardless of its underlying type. The details of working with container formats and codec types are left to the framework, leaving you to focus on how you want to use those assets in your app. Second, AVAsset provides a level of independence from the media’s location. You create an asset instance by initializing it with the media’s URL. This could be a local URL, such as one contained within your app bundle or elsewhere on the file system, or it could also be a resource such as an HLS stream hosted on a remote server. In either case, the framework performs the work necessary to efficiently retrieve and load the media in a timely manner on your behalf. Removing the burdens of dealing with media formats and locations greatly simplifies working with audiovisual media.

AVAsset is a container object composed of one or more instances of AVAssetTrack, which models the asset’s uniformly typed media streams. The most commonly used track types are audio and video tracks, but AVAssetTrack also models other supplementary tracks such as closed captions, subtitles, and timed metadata (see Figure 3-1).

You retrieve an asset’s collection of tracks using its tracks property. In many cases, you’ll want to perform operations on a subset of an asset’s tracks rather than on its complete collection. In these cases, AVAsset also provides methods to retrieve subsets of tracks based on criteria such as identifier, media type, or characteristic.

Creating an Asset

You create an AVAsset by initializing it with a local or remote URL pointing to a media resource, as shown in the following example:

let url: URL = // Local or Remote Asset URL

let asset = AVAsset(url: url)

AVAsset is an abstract class, so when you create an asset as shown in the example, you’re actually creating an instance of one of its concrete subclasses called AVURLAsset. In many cases this is a suitable way of creating an asset, but you can also directly instantiate an AVURLAsset when you need more fine-grained control over its initialization. The initializer for AVURLAsset accepts an options dictionary, which lets you tailor the asset’s initialization to your particular use case. For instance, if you’re creating an asset for an HLS stream, you may want to prevent it from retrieving its media when a user is connected to a cellular network. You could do this as shown in the following example:

let url: URL = // Remote Asset URL

let options = [AVURLAssetAllowsCellularAccessKey: false]

let asset = AVURLAsset(url: url, options: options)

Passing a value of false for the AVURLAssetAllowsCellularAccessKey option indicates that you want this asset to retrieve its media only when a user is connected to a Wi-Fi network. See AVURLAsset Class Reference for information about its available initialization options.

Preparing Assets for Use

You use the properties of AVAsset to determine its features and capabilities, such as its suitability for playback, duration, creation date, and metadata. Creating an asset does not automatically load its properties or prepare it for any particular use. Instead, the loading of an asset’s property values is deferred until they are requested. Because property access is synchronous, if the requested property hasn’t previously been loaded, the framework may need to perform a significant amount of work to return a value. In macOS, this can result in an unresponsive user interface if an unloaded property is accessed from the main thread. In iOS and tvOS, the situation can be even more serious because media operations are performed by the shared media services daemon. If the request to retrieve an unloaded property value is blocked for too long, a timeout occurs resulting in a termination of media services. To prevent this from happening, load an asset’s properties asynchronously.

AVAsset and AVAssetTrack adopt the AVAsynchronousKeyValueLoading protocol, which defines the methods you use to query the current loaded state of a property and asynchronously load one or more property values, if needed. The protocol defines two methods:

public func loadValuesAsynchronously(forKeys keys: [String], completionHandler handler: (() -> Void)?)

public func statusOfValue(forKey key: String, error outError: NSErrorPointer) -> AVKeyValueStatus

You use the loadValuesAsynchronouslyForKeys:completionHandler: method to asynchronously load one or more property values. You pass it an array of keys, which are the names of the properties to load, and a completion block that is called after a status is determined. The following example shows how to asynchronously load an asset’s playable property:

// URL of a bundle asset called 'example.mp4'

let url = Bundle.main.url(forResource: "example", withExtension: "mp4")!

let asset = AVAsset(url: url)

let playableKey = "playable"

// Load the "playable" property

asset.loadValuesAsynchronously(forKeys: [playableKey]) {

    var error: NSError? = nil

    let status = asset.statusOfValue(forKey: playableKey, error: &error)

    switch status {

    case .loaded:

        // Sucessfully loaded. Continue processing.

    case .failed:

        // Handle error

    case .cancelled:

        // Terminate processing

    default:

        // Handle all other cases

    }

}

You check the property’s status in the completion callback using the statusOfValueForKey:error: method. A status of AVKeyValueStatusLoaded indicates the property value was successfully loaded and can be retrieved without blocking. A status of AVKeyValueStatusFailed indicates the property value is not available due to an error encountered while attempting to load its data. You can determine the reason for the error by inspecting the NSError pointer. In all cases, be aware that the completion callback is invoked on an arbitrary background queue. Dispatch method invocations back to the main queue before performing any user interface-related operations.

Working with Metadata

Media container formats can store descriptive metadata about their media. As a developer, it’s often challenging to work with metadata, because each container format has its own unique metadata format. You typically need a low-level understanding of the format to read and write a container’s metadata, but AVFoundation simplifies working with metadata through the use of its AVMetadataItem class.

In its most basic form, an instance of AVMetadataItem is a key-value pair representing a single metadata value such as a movie’s title or an album’s artwork. In the same way that AVAsset provides a normalized view of your media, AVMetadataItem provides a normalized view of its associated metadata.

let url = Bundle.main.url(forResource: "audio", withExtension: "m4a")!

let asset = AVAsset(url: url)

let formatsKey = "availableMetadataFormats"

asset.loadValuesAsynchronously(forKeys: [formatsKey]) {

    var error: NSError? = nil

    let status = asset.statusOfValue(forKey: formatsKey, error: &error)

    if status == .loaded {

        for format in asset.availableMetadataFormats {

            let metadata = asset.metadata(forFormat: format)

            // process format-specific metadata collection

        }

    }

}

let metadata = asset.commonMetadata

let titleID = AVMetadataCommonIdentifierTitle

let titleItems = AVMetadataItem.metadataItems(from: metadata, filteredByIdentifier: titleID)

if let item = titleItems.first {

    // process title item

}

// Collection of "common" metadata

let metadata = asset.commonMetadata

// Filter metadata to find the asset's artwork

let artworkItems =

    AVMetadataItem.metadataItems(from: metadata,

                                 filteredByIdentifier: AVMetadataCommonIdentifierArtwork)

if let artworkItem = artworkItems.first {

    // Coerce the value to an NSData using its dataValue property

    if let imageData = artworkItem.dataValue {

        let image = UIImage(data: imageData)

        // process image

    } else {

        // No image data found

    }

}

Playing Media

The asset model described in the previous section is the cornerstone of the playback use case. Assets represent the media you want to play, but are only part of the picture. This section discusses the additional objects needed to play your media and shows how to configure them for playback (see Figure 3-2).

class PlayerViewController: UIViewController {

    @IBOutlet weak var playerViewController: AVPlayerViewController!

    var player: AVPlayer!

    var playerItem: AVPlayerItem!

    override func viewDidLoad() {

        super.viewDidLoad()

        // 1) Define asset URL

        let url: URL = // URL to local or streamed media

        // 2) Create asset instance

        let asset = AVAsset(url: url)

        // 3) Create player item

        playerItem = AVPlayerItem(asset: asset)

        // 4) Create player instance

        player = AVPlayer(playerItem: playerItem)

        // 5) Associate player with view controller

        playerViewController.player = player

    }

}

Observing Playback State

AVPlayer and AVPlayerItem are dynamic objects whose state changes frequently. You often want to take actions in response to these changes, and the way you do so is by using Key-Value Observing (KVO) (see Key-Value Observing Programming Guide). With KVO, an object can register to observe another object’s state. When the observed object’s state changes, the observer is notified with details of the state change. Using KVO makes it easy for you to observe state changes to AVPlayer and AVPlayerItem and take actions in response.

One of the most important AVPlayerItem properties to observe is its status. The status indicates if the player item is ready for playback and generally available for use. When you first create a player item, its status has a value of AVPlayerItemStatusUnknown, meaning its media hasn’t been loaded or been enqueued for playback. When you associate a player item with AVPlayer, it immediately begins enqueuing the item’s media and preparing it for playback. The player item becomes ready for use when its status changes to AVPlayerItemStatusReadyToPlay. The following example shows how you can observe this state change:

let url: URL = // Asset URL

var asset: AVAsset!

var player: AVPlayer!

var playerItem: AVPlayerItem!

// Key-value observing context

private var playerItemContext = 0

let requiredAssetKeys = [

    "playable",

    "hasProtectedContent"

]

func prepareToPlay() {

    // Create the asset to play

    asset = AVAsset(url: url)

    // Create a new AVPlayerItem with the asset and an

    // array of asset keys to be automatically loaded

    playerItem = AVPlayerItem(asset: asset,

                              automaticallyLoadedAssetKeys: requiredAssetKeys)

    // Register as an observer of the player item's status property

    playerItem.addObserver(self,

                           forKeyPath: #keyPath(AVPlayerItem.status),

                           options: [.old, .new],

                           context: &playerItemContext)

    // Associate the player item with the player

    player = AVPlayer(playerItem: playerItem)

}

The prepareToPlay method registers to observe the player item’s status property using the addObserver:forKeyPath:options:context: method. Call this method before associating the player item with the player to make sure you capture all state changes to the item’s status.

To be notified of status changes, you implement the observeValueForKeyPath:ofObject:change:context: method. This method is invoked whenever the status changes, giving you the chance to take some action:

override func observeValue(forKeyPath keyPath: String?,

                           of object: Any?,

                           change: [NSKeyValueChangeKey : Any]?,

                           context: UnsafeMutableRawPointer?) {

    // Only handle observations for the playerItemContext

    guard context == &playerItemContext else {

        super.observeValue(forKeyPath: keyPath,

                           of: object,

                           change: change,

                           context: context)

        return

    }

    if keyPath == #keyPath(AVPlayerItem.status) {

        let status: AVPlayerItemStatus

        if let statusNumber = change?[.newKey] as? NSNumber {

            status = AVPlayerItemStatus(rawValue: statusNumber.intValue)!

        } else {

            status = .unknown

        }

        // Switch over status value

        switch status {

        case .readyToPlay:

            // Player item is ready to play.

        case .failed:

            // Player item failed. See error.

        case .unknown:

            // Player item is not yet ready.

        }

    }

}

The example retrieves the new status value from the change dictionary and switches over its value. If the player item’s status is AVPlayerItemStatusReadyToPlay, then it’s ready for use. If a problem is encountered while attempting to load the player item’s media, the status is AVPlayerItemStatusFailed. If a failure occurred, you can retrieve the NSError object providing the details of the failure by querying the player item’s error property.

Performing Time-Based Operations

Media playback is a time-based activity—you present timed media samples at a fixed rate over a certain duration. Time-based operations, such as seeking through the media, play a central role when building media playback apps. Many of the key features of AVPlayer and AVPlayerItem are related to controlling media timing. To learn use these features effectively, you should understand how time is represented in AVFoundation.

Several Apple frameworks, including some parts of AVFoundation, represent time as a floating-point NSTimeInterval value that represents seconds. In many cases, this provides a natural way of thinking about and representing time, but it’s often problematic when performing timed media operations. It’s important to maintain sample-accurate timing when working with media, and floating-point imprecisions can often result in timing drift. To resolve these imprecisions, AVFoundation represents time using the Core Media framework’s CMTime data type.

Core Media is a low-level C framework that provides much of the functionality found in AVFoundation and higher-level media frameworks on Apple platforms. In most cases you’ll work with Core Media through the higher-level interfaces provided by AVFoundation, but a commonly used data type it provides is called CMTime.

public struct CMTime {

    public var value: CMTimeValue

    public var timescale: CMTimeScale

    public var flags: CMTimeFlags

    public var epoch: CMTimeEpoch

}

This struct defines a rational—or fractional—representation of time. The two most important fields defined by CMTime are its value and timescale. CMTimeValue is a 64-bit integer defining the numerator of the fractional time, and CMTimeScale is a 32-bit integer defining the denominator. This struct makes it easy to represent times expressed in terms of the media’s frame rate or sample rate.

// 0.25 seconds

let quarterSecond = CMTime(value: 1, timescale: 4)

// 10 second mark in a 44.1 kHz audio file

let tenSeconds = CMTime(value: 441000, timescale: 44100)

// 3 seconds into a 30fps video

let cursor = CMTime(value: 90, timescale: 30)

Core Media provides a number of ways to create CMTime values and perform arithmetic, comparison, validation, and conversion operations on them. If you're using Swift, Core Media also adds a number of extensions and operator overloads to CMTime, making it easy and natural to perform many common operations. See Core Media Framework Reference for more information.

var player: AVPlayer!

var playerItem: AVPlayerItem!

var timeObserverToken: Any?

func addPeriodicTimeObserver() {

    // Notify every half second

    let timeScale = CMTimeScale(NSEC_PER_SEC)

    let time = CMTime(seconds: 0.5, preferredTimescale: timeScale)

    timeObserverToken = player.addPeriodicTimeObserver(forInterval: time,

                                                       queue: .main) {

        [weak self] time in

        // update player transport UI

    }

}

func removePeriodicTimeObserver() {

    if let timeObserverToken = timeObserverToken {

        player.removeTimeObserver(timeObserverToken)

        self.timeObserverToken = nil

    }

}

var asset: AVAsset!

var player: AVPlayer!

var playerItem: AVPlayerItem!

var timeObserverToken: Any?

func addBoundaryTimeObserver() {

    // Divide the asset's duration into quarters.

    let interval = CMTimeMultiplyByFloat64(asset.duration, 0.25)

    var currentTime = kCMTimeZero

    var times = [NSValue]()

    // Calculate boundary times

    while currentTime < asset.duration {

        currentTime = currentTime + interval

        times.append(NSValue(time:currentTime))

    }

    timeObserverToken = player.addBoundaryTimeObserver(forTimes: times,

                                                       queue: .main) {

        // Update UI

    }

}

func removeBoundaryTimeObserver() {

    if let timeObserverToken = timeObserverToken {

        player.removeTimeObserver(timeObserverToken)

        self.timeObserverToken = nil

    }

}

// Seek to the 2 minute mark

let time = CMTime(value: 120, timescale: 1)

player.seek(to: time)

// Seek to the first frame at 3:25 mark

let seekTime = CMTime(seconds: 205, preferredTimescale: Int32(NSEC_PER_SEC))

player.seek(to: seekTime, toleranceBefore: kCMTimeZero, toleranceAfter: kCMTimeZero)