Audio Unit Extensions

DOUG WYATT: Good morning.

I'm Doug Wyatt from the Core Audio team and I would like to show you something new we have been working on called Audio Unit Extensions. This is a new technology in iOS 9 and OS X El Capitan.

About Audio Units: we've had had this technology in our operating systems since the beginning OS X and iOS. The operating system includes a great number of built-in units ranging from I/O units and mixers, a lot of different effects ranging to software sampler.

We use these internal Audio Units in many of our higher-level APIs, for example, the media playback stack.

But Audio Units are also a widely adopted third-party plug-in format on OS X. There are literally thousands of third-party Audio Units in the market out there. Now, Audio Unit extensions for the first time bring us a full plug-in model on both OS X and iOS. It is built on top of the app extension technology, which means if you are writing plug-ins you can package them into apps, and those apps can be sold on the App Stores.

As part of this technology, we have modernized the API and yet at the same time maintained compatibility, and in this session I will go through details of this new API, which we are calling the version 3 Audio Unit API.

It's based on an Objective-C class called AUAudioUnit that's in the Audio Unit framework. And as an Objective-C class, of course, it plays nicely with Swift, as we will see. In this session, we are also going to look at a number of classes in the AVFoundation framework. We have AV Audio Unit component manager and AV Audio Unit component. These are used to located the audio components on the system. Those appear for the first time in iOS 9. They also exist on Yosemite.

And we will also be using AVAudioEngine in some of our example code we will be showing today, in particular the AVAudioUnit class and AVAudioUnitEffect class. Those have been available since last year's OS releases.

So about compatibility now. This is how things look now in OS X. We have our existing version 2 Audio Unit hosts and existing version 2 Audio Unit implementations. The hosts start their communication with audio component instance new, and our implementations are built on audio component factory functions.

We have a new set of APIs here, so we will have new hosts using those new APIs. And new Audio Units implemented using those new APIs. Hosts will communicate with the class AU Audio Unit. New version 3 Audio Units will subclass AU Audio Unit.

So that's two separate APIs. What are we going to do to be compatible? We have built bridges between these two APIs.

So thanks to these bridges, we will find that new version 3 hosts should be almost completely compatible with existing version 2 Audio Units. And conversely, existing version 2 hosts would only need minor source changes to work with new version 3 Audio Units. And I will detail those API changes in a little bit.

So now I would like to give you a demo of a new Audio Unit working in an only slightly modified version of Logic Pro. I have a little session here, it has a drum loop built in. And here I'm going to apply an Audio Unit to this track.

So here are all of the Apple built-in Audio Units.

And here I have a new demo Audio Unit called v3 Distortion.

So I can open this Audio Unit.

I can find the preset I like, and we can hear Logic playing through this Audio Unit. There it's dry. Completely distorted.

Now, if I go to activity monitor here, we can see that this distortion Audio Unit is running in a separate process, AU v3 distortion. It's consuming a little bit of CPU. It has some threads running.

Now, suppose this Audio Unit has a bug in it, and it crashes. Well, I can simulate that here in activity monitor. I can force quit it. And notice in Logic, the view went blank but the music kept playing.

So here is a diagram of what we were just looking at. That's a slightly modified version of Logic Pro, but it's still basically communicating using the existing version 2 API, which is bridged to AU Audio Unit and in turn, in that separate extension service process, we saw the distortion units AU Audio Unit subclass running along with its custom View Controller. In the Logic process, there is also a View Controller, and you see how these are bridged across the process boundary.

Now, I'd like to talk about hosting Audio Units and I will show you an example that uses the version 3 APIs. We have sample code called Audio Unit v3 Example. I checked a couple of hours ago, but it hadn't appeared yet. I hope it comes out today. In this sample code project, you will see there are a number of targets and one of them is called AU Host.

Now, this application is fairly simple and straightforward, but it shows how to find and open Audio Units that are on the system, how to connect them together into a rendering chain, how to select Audio Unit presets, and how to open an Audio Unit's custom view.

So in the AU host app, we have something called simple play engine, which is a Swift class that uses AVAudioEngine. It uses an AV audio player node connected to an AV Audio Unit effect. That AV Audio Unit effect in turn exposed an underlying AU Audio Unit, which is the maiden class of the version 3 Audio Unit API.

We have the player to the effect, to the mixer, to the output. That's how the simple play engine makes sound.

We will also see how to use the AV Audio Unit component manager class to select from the AV Audio Unit components on the system and use that to control what kind of AV Audio Unit effect gets selected. So let's get into a little bit of code, but first there is a very fundamental data structure here when working with Audio Units. We have the audio component description and its first three fields: the component type, type, subtype, and manufacturer. That tuple uniquely identifies an Audio Unit in the system. The flags are also important. They are partially populated by the audio component, and there are new ones populated by the system. We will describe some of those as we go along. The important thing here is this is the key that identifies the plug-in.

So to find Audio Unit components on the system, the first thing we do is create an audio component description that contains a wildcard. Here we say the component type is effect. That's not a wildcard, but we have component subtype and manufacturer of zero. Those are wildcards, so we have built a component description here that identifies any effect.

And then we can take that any effect component description and pass it to AV Audio Unit component manager, and it will give us back all of the effects on the system matching that wildcard.

So here we get an array of AV Audio Unit component objects, and those contain things like the name, tags, also the audio component description of that unit.

So here we have got an array of components. We can pass that back to the UI, and in turn the UI can call this method in the simple play engine to select one of these previously vended effect components. So here it gives us a component. We are going to fetch the audio component description out of that, pass it to an internal method, and the guts of that internal method is here. We are going to call a new class method of AV Audio Unit.

And this method asks it to create an instance based on the component description we have here.

Now, this is an asynchronous function, meaning it's going to go off and start instantiating it, and then it's going to call this closure when it actually has instantiated the Audio Unit and we are ready to use it.

So here we are in our callback. This is Swift closure syntax. We have our AV Audio Unit.

And then we can attach it to our engine. We have stored it into a member variable, the effect.

And now we have an AV Audio Unit know that's the effect. We are going to patch that into the engine. We will disconnect the effect from the main mixer, then connect from the player to the effect.

And then from the effect to the main mixer node.

So now we have got an effect in our play engine.

And now we can store the actual AU Audio Unit. That's the plug-in, and here we can do all kinds of interesting things like manipulate the component's effect, presets, and parameters.

For instance, here, we will just get the list of factory presets.

And this too can populate a field in the table view -- rather, in the UI.

So the user can choose the factory preset.

And finally, I would like to show you how in the app I will show you in a minute, we can get the Audio Unit's custom view and embed it into the host application's view. Here we are in the View Controller of the host, so we are going to ask the play engine, give me your Audio Unit, and then we are going to ask the Audio Unit for a View Controller. When it's done with that, it will call us back with a View Controller that we can embed in the host's view.

Okay. I would like to bring up my colleague Michael Hopkins now to show you this app actually running now.

MICHAEL HOPKINS: Thank you very much, Doug. I'm delighted to have this opportunity today to show you this AVAudioEngine-based v3 host application running on an iPad here.

As you can see, I'm going to launch that by tapping the icon for the host.

And on the left-hand side of the screen we have a list of all of the effects Audio Units that are present on the system. And this includes both the built-in Apple audio component-based effects as well as several new extension-based v3 Audio Units I installed myself.

At the top of the screen, I have a Play button that I can tap to toggle the playback of a drum loop.

Now, let's see how I can apply some effect nodes and add them to the graph. First, I will play this with no effect and then I will add a couple effects so you can hear that working.

With the high pass filter, it's filtering out almost all of the sounds of the cymbals and other higher frequencies.

A delay, which is a little bit hard to hear in this room. And I'm going to go ahead and stop that. So now I would like to show you for the first time an extension-based Audio Unit running on this iPad. That is the distortion demo.

When I select that, now you can see the list of all of the factory presets that the Audio Unit is publishing.

These include some drum-specific ones as well as some really crazy, wild effects like alien chatter.

Now, as Doug mentioned , v3 Audio Unit can have a custom view on iOS. And I'm going to show you that. I'm going to go ahead and tap the View button. And what we have done is we have loaded that View Controller from the Audio Unit and I have installed it as a child View Controller within our application context. So for the first time, we have a built in Audio Unit with a UI running in our host.

We have a large slider, excuse me, a large knob that I can use to control the amount of distortion.

And let me go ahead and play that for you so you can hear that in action.

It's really a lot of fun. It's an amazing experience to be able to have that Multi-Touch UI working fluidly in a host application without having to go through all of the hassle of switching out to another application, doing some tweaks, switching back to your host, starting recording, switching back. Now, you won't have to do that ever again.

Thank you.

And I'd also like to point out further that this is the same Audio Unit that you saw running in Logic in Doug's earlier demo. In fact, the source code for the Audio Unit is identical. No changes were required. The drawing code is also very similar because I chose to write this using Core Animation so that API is almost fully portable. The only changes that were necessary to bring this Audio Unit to iOS are in the event model, whereas we have had to use the touch events in UIKit versus the AppKit mouse events on the desktop. So really you guys have an opportunity to with only a few changes to publish an Audio Unit both on the desktop and on iOS. Thank you very much. Back to you, Doug.

DOUG WYATT: Thank you, Michael.

So I would like to talk about using Audio Units in your host applications in situations where you are not using AVAudioEngine.

We have a similar method on the AU Audio Unit class to asynchronously create an instance of the component description. You see that there.

We also, for those of you with existing version 2 hosts, a minimal translation path is to start using audio component instantiate. We will talk about that in detail in a bit.

Now, I would like to talk about the subject of extension service processes versus plug-ins loaded into host processes.

Now, as anybody who has worked with Audio Units is aware, of course, with our existing plug-in model, the plug-ins are always loaded into the host's progress, and this remains true for version 3 hosts. If it's a version 2 existing plug-in, and that might be one of the Apple built-in ones on iOS or it could be a third-party one on OS X, but in any case if it's a version 2 Audio Unit, regardless of any other factor, it's always in the host's process.

Now, version 3 Audio Units have a slightly more complicated story here. By default, version 3 Audio Units are loaded into a separate extension service process. And this is the diagram we saw before with Logic.

This is true, again, whether it's a version 2 host or a version 3 host.

Now, on OS X only it is possible for the plug-in to be loaded directly into the host's process. Now, for this to happen, both parties have to opt in. The host when instantiating the Audio Unit has to pass this option to any of the asynchronous creation methods we just saw, and you see the name of that new flag there called Load in Process. And the Audio Unit also has to be packaged specially and consent to this with a plist entry called Audio Component Bundle. So if both parties do opt in, then the framework will actually load the plug-in into the host's process. So the host will be communicating directly with the plug-in's AU Audio Unit subclass.

Now, as a host author, why would you want to do this? There is a tradeoff here between safety and performance is the reason. Of course, it's a security risk to be loading third-party code into your app, and if it crashes inside your app, then users might blame you instead of the misbehaving plug-in.

But on the other hand, we have performance reasons where you may want to load plug-ins into your process if you are a host, because there is some overhead to communicating with that separate extension service process. And we have measured that as being on the order of 40 seconds microseconds per render cycle , o you can do the math to figure out how significant that is in the context of your host. You have some number of out-of-process plug-ins you might be communicating with, so you have to add that up. And there is also the factor of how much audio you are asking them to render.

For example, if you are rendering at a very low latency of 32 frames, that's a 1 millisecond render interval, so overhead of 40 microseconds could be significant at 5.5 percent. So that's your tradeoff if you are a host author.

I mentioned earlier that existing version 2 Audio Unit hosts need a few changes to work with version 3 Audio Units, and here is what has to change.

I mentioned the audio component description flags, and in there, the component flags. There is a new flag called Requires Async Instantiation. That's set for most if not all new version 3 Audio Units, so if you see that flag set in the component description, you have to use the new Audio Component Instantiate method instead of Audio Component Instance New.

Now, similarly in an existing v ersion 2 host, if you want to access an Audio Unit's View Controller, then you also need to use a new asynchronous method to do that. There is a new property, Request View Controller. It is also asynchronous. You can read about the details of that in Audio Unit properties.h.

So about these asynchronous methods. You can use the new methods with version 2 units, but you must use them with version 3 units when the flags are set. And the reasoning here, well, the big sort of externally facing reason is that it helps responsiveness. If it's going to take half a second to instantiate that Audio Unit, well, if you unblock the main thread, your host applications, meters, or other animations will keep drawing smoothly.

Now, especially when updating existing code -- and this was the first thing I did when testing internal test code -- it's tempting to sit there and wait on the main thread for the asynchronous operation to complete. Well, don't do that, because not only will you block any graphics you are doing, but you will also block some underlying operations in the framework that are actually required for the Audio Unit to be instantiated. So you will deadlock if you block on the main thread. Don't do that. Now, I would like to switch gears from talking about hosting Audio Units to creating Audio Units with the new version 3 API.

First, a few words about app extensions since the new Audio Unit model is based on app extensions.

App extensions are bundles with a file type extension of .appex. Xcode will build them into an app's plug-ins directory, and we saw how they get loaded by the system into separate extension service processes. You can read all about the nuts and bolts of app extensions in the app extension programming guide.

Now, our new sample code project, Audio Unit v3 Example, contains a sample Audio Unit implementation called Filter Demo.

Filter Demo, when you look at that sample project, has three targets. It has what we call the containing app, and what it contains is the app extension as well as a framework where a lot of its common code exists. Both the app and the extension link against this framework.

Now, inside this framework, we have two main classes. There is AU v3 Filter Demo, that's the AU Audio Unit subclass, and the Filter Demo View Controller that controls the custom view of the Audio Unit. Now, what's cool about doing things this way is that while we are developing our signal processing and view code, we can do this all in the context of our own app so we are debugging not in a separate SPC service process, but we are debugging and developing our code right there interactively in our own app.

We also let our app look like something when the user opens it. It's not just a plug-in for somebody else. And we are not duplicating any code to be able to accomplish that.

There is one extra bonus here that on OS X, if we want to, we can designate this framework as being the bundle that a host process can load into itself. So let's look at the app extension. It has an info plist with important entries. The NSExtensionPointIdentifier tells the system what kind of extension it is, the main storyboard tells the system, when you launch my extension service process, open the storyboard.

And finally, there is an Audio Components array that tells the system, here are the audio component descriptions that I am registering.

Just a quick reminder here, in your storyboard, you will want to be sure to specify your custom class. You may need to specify the module if you are building it into a separate framework like we are here.

Then the extension actually has no code in it other than this little bit of dummy code to keep it from being empty. We have to link against the Filter Demo framework. All of the good stuff is in there, and here we just have a global variable referring to it. Let's move onto the framework now.

So the main class in the framework is the Filter Demo View Controller. In extension terminology, this is the extension's principal class. Whenever the extension is created or loaded, the system will create an instance of that principal class. And it's got two main jobs. It in turn creates the AU Audio Unit subclass and, as a View Controller as you would expect, it creates and manages the plug-ins custom view.

Here is the class declaration for the Filter Demo View Controller. It derives from AU View Controller, which is either an NS or UI View Controller basically, and it also implements a protocol called AU Audio Unit factory. That's a simple protocol and implements exactly one method, Create Audio Unit with Component Description.

And the job of this method is to create the AU Audio Unit subclass.

And here it is, the AU v3 Filter Demo.

Now, let's look at that AU Audio Unit subclass. So for reasons we will get into in a bit, these are actually embedded C++ classes or objects. The filter DSP kernel is where all of the math happens.

We will listen to it later. It's a little more interesting than looking at its code. We have some code here dealing with the buses. This is an effect. It has one input and one output, and our base class is going to want us to provide arrays of buses so we have numbers to support that. And we have something called a parameter tree. We will see what that is in just a second. Here is the initializer.

So the first thing we do is initialize our input and output buses, and then we wrap them in bus arrays.

And these arrays each contain a single bus.

And now we are looking at parameters. So every parameter is an object, and you can think of this object as kind of the bridge between your implementation and the host. In the middle, there is the parameter object. This is a simple low-pass filter, so it's only got two parameters, a cutoff frequency and residence.

Every parameter has an identifier, so here we are saying it's cutoff. It has a localizalbe name. We are being bad and not localizing it here.

It has an address. We will talk about that in a bit. Arrange, and some units, and flags which you will recognize as being almost identical to what we do with version 2 Audio Units. So here we have created our first parameter. We will create our second one almost identically.

And then finally we can create our parameter tree, passing an array of those two parameters.

Now, we have our parameter tree, and we want to wire it up so that it's connected to our DSP code.

And the way we do this is install a block into the parameter tree called the Implementer Value Observer.

So this block will get called any time somebody, whether it's the host or our own view, changes a parameter, And so in response to that change, we will simply set that new value on our filter DSP kernel so that it takes immediate audible effect.

Now, in the other direction, there are times when the tree needs to refresh its value based on what we have in our signal processing. That's what this block does. It fetches the current value from the DSP and returns it to the tree.

Next, this is an important override method. If you're familiar with the version 2 Audio Unit API, this was called Audio Unit Initialize, which is not a good name choice in the Objective-C world. So we decided to make it very specific.

What was initialize time is really prepare to render and allocate the resources that are associated with rendering.

So there are things like buffers, DSP state, and so on.

So the first thing we do here is called the base class method.

Then we can ask our input bus to allocate some memory for audio input to the plug-in, and we can initialize our signal processing code here based on the current channel count and sample rate of the output bus.

So entirely parallel, we have a method that's called to deallocate render resources. And here, too, we call the base class and basically undo whatever we did when allocating.

So the process of rendering works through a block that gets called every render cycle, but we are asked to provide this block before starting to render.

Here we are going to capture our C++ members into local variables that are pointers. Now, the reason for this is that we are going to be running our block for rendering in a real-time context where it's not safe to access any Objective-C objects because the runtime could block and cause an audio glitch. So, again, we are just going to capture our C++ member variables.

And then we can return the block.

It returns AU Audio Unit status.

And if you are familiar with the version 2 API, the parameters are largely the same. There is a time stamp, a number of sample frames, an output audio buffer list, and here is something new called the real-time event list head.

I will talk about that in detail, but it relates to scheduled parameters and MIDI events.

And finally, the Pull Input block. This is how the host tells us, the implementer of the Audio Unit, where to get input from.

So in the guts of the input block, the first thing we will do is pass that Pull Input block to our input C++ object and ask that input object to fetch the audio input for this render cycle.

Then we are going to do some housekeeping with buffers. We will send them down to the DSP state, and finally, we ask the DSP state to process the audio for this render cycle.

It's already been informed of the buffers, and we are just going to give it a time stamp, the frame count, and the linked list of real-time events. So that's the guts of this Audio Unit, there is not a whole lot of code. There is a lot more code that does the actual signal processing, but as I mentioned before, it's better to listen to that than to look at it. So I would like to bring Michael Hopkins back up to show us the AU v3 Filter Demo.

MICHAEL HOPKINS: Thank you, Doug. I'm going to go ahead and start with the app container that contains the extension.

You will see first on the screen in the left-hand side is our Filter Demo, which we have distributed as sample code. To the right of it is the distortion demo application that I showed you earlier. I will go ahead and launch the Filter Demo.

Now, at the top of the screen, you will notice the two parameters that Doug talked about. We have the cutoff in the residence parameter, and these are represented in our UI with a slider and a text field. And this portion of the UI is actually contained in the application, whereas the larger area in the main screen with the graph is actually our embedded view from the Audio Unit. I can go ahead and change the value of the parameters by dragging the slider.

And what's happening here is the application is changing the value of that parameter. And the view is listening for changes to that parameter, and then it's updating. As you will see, that update is live. Conversely, I can interact directly with our embedded Audio Unit view by tapping in the graph and dragging. And you will notice that as I drag that with my finger, the application is receiving notifications that the parameters have changed and it's updating in turn. But that's kind of a boring demo without any audio going through it, wouldn't you say? Let's go ahead and take a listen to that.

I could do this all day. You got time? Now, it's really, really cool to just, the fluidity. Thank you. Just how fun it is to use your fingers to just be able to play with that in a Multi-Touch UI. It's phenomenal. Another thing that's cool about this is because we have designed the user interface in such a way that it can adapt to any size that it's embedded in, we can actually take this iPad and we can rotate it sideways, and you can see that now the user interface is updated. And from a portrait view to a landscape view, and vice versa.

Now, we are doing that because we are supporting Auto Layout and we are looking at size classes, so that's really great. But what happens when we go and we take this and we put it into our host app, which has a much smaller amount of screen real estate dedicated to plug-ins? So I will go ahead and switch back, and we are going to open the host there. I'm going to get rid of the beautiful distortion demo and embed our Filter Demo view. Tap on View to load that. And now you can see that that's being loaded in a constrained vertical space and a very wide horizontal space yet it still works as you would expect, and none of the labels overlap. It still works exactly as we would expect.

So this is a fantastic new technology, and we are so excited to be able to finally bring it to you guys and I can't wait to see what do in your own iOS apps. Thank you. DOUG WYATT: Thank you, Michael.

Just a few words now about the containing app in general. It is a vehicle for your plug-in and helps you with rapid iteration and development, but you can think about putting some extra things in that containing app. We saw with the Filter Demo that it has the simple play engine, you may for whatever reason want a more complex play engine of some sort. This is also a good place, the containing app, to try putting creative touch controller. There may not be room in a plug-in view for your full touch controller. Maybe there is, but you might think about having extra, an extra-large version or something in your containing app.

The app is also a good place for any documentation or help that would make the plug-in view a little dense.

A few final words about creating an app extension here. So if you are going to build a framework to be loaded in process on OS X, despite what we are doing here with Swift, we can't recommend that you do this on OS X because the Swift API is subject to change. If you build your plug-in against one version of the Swift runtime and you are loaded into a host that happens to be using another version, there could be collisions, and that would be bad.

We realize when you look at the sample code here and you try to build your own plug-ins, there is a lot, you know, there is three related targets that have to be built properly. It's a little bit complicated. We do plan in Xcode's template, but for now you can copy liberally from the Filter Demo.

Okay. Now I would like to talk in general about the modernized AU Audio Unit API from both the host and the implementation sides. I would like to compare the way that properties are handled in version 2 versus version 3. In version 2 Audio Unit API, we have scope- and element-based properties. A large number of properties are in the global scope so you have a bunch of code where you type. K Audio Unit scope global, element 0.

And it's painful, especially from Swift, where we have property values that are void pointers. You end up typing unsafe mutable pointer all over the place, and that makes my head hurt. We have these functions with long argument lists.

And by comparison in the version 3 API, well, properties are properties.

We use a dot syntax in Objective-C and Swift, so you can write AU.maximum frames to render.

We also implement our classes to be key-value coding and key-value observing compliant so you can use value for key, and add observer for key path.

We also added a special KVO method to the bus array, add observer to all buses, so you don't have to simultaneously be watching for buses to come and go just so that you can add KVO observers on them. That can be a painful cycle to chase.

Speaking of buses, they are full-fledged objects in the new API. We have the AU Audio Unit bus array. The AU Audio Unit has an array of input buses, an array of output buses.

And the buses have two major properties. They have a format and a name. The format is manipulated by the host. We are able to reject formats that we don't like. We use the same formats in version 3 Audio Units as version 2.

Let's look at parameters. We have some of the same problems here in the version 2 API with parameters as we did with properties. We have these unwieldy scope element ID tuples. And furthermore, in some very complex AUs we just didn't have enough bits.

We have these functions with long argument lists again, and we also have a complicated AU event listener API.

In the version 3 API, I hinted at this earlier with the parameter tree and the Filter Demo. Well, it is a full tree. Parameters can be grouped, and here we have an example of a simple analog synthesizer emulation. It has groups for oscillator, filter, and amplifier. The filter and amplifier have envelope groups beneath them. And the most brightly colored boxes beneath are the parameters. So the waveform octave, filter cutoff and resonance, and the envelope attack, sustain, and release.

So these boxes are all nodes, whether they are groups or parameters, and every node in the parameter tree has a unique and permanent ID. This is like a C identifier.

So using these unique IDs, we can use KVC to go and find a parameter we are looking for, such as oscillator.wave or filter.envelope.attack. And this would be a lot more flexible for these very complex audio units that have very large parameter trees.

Now, you will notice that parameters have numeric addresses and that they are 64 bits, but we do have to treat them as transient in any situation where we aren't the one who made up that addressing scheme.

So that means if I'm a host application and I want to record some parameter automation, I should record that automation using the key value path -- I'm sorry, the key path of that parameter and not its address. I alluded to this earlier. The AU parameter object is the focus of communication for parameter values between hosts and views, and on the other side, the Audio Unit implementations.

Now, from the host's point of view, the parameter object has properties like its value, also a minimum and maximum value, and so on. So we can set and get parameter values using dot notation as you would expect.

Now, we can also set values in such a way to prevent a feedback loop, which is useful both for performance and for keeping the UI smooth. We don't want to be getting notifications that are slightly different from what we are doing as we move a slider. So the set value method accomplishes that.

And that token is obtained from a method to add an observer to a parameter, or its tree, or a group in the tree.

And when we do that, then we can get called back from the parameter. That's what we see at the bottom there. There is the block called the AU parameter observer, and it passes us the address and the new value of the parameter that changed. As for the implementation, we saw this in the Filter Demo. It has the implementer value observer and value provider blocks. Now, in Filter Demo, it installed these blocks on the tree. It's also possible to install them at any level of the tree, even on individual parameters.

I would also like to show off what we have done with parameter scheduling because I think this is a big improvement over the version 2 API in this area.

We have the host and the implementation dealing with things somewhat separately, but here we have the AU Audio Unit base class doing some help, it's helping us implement this. So the host can obtain from the AU Audio Unit a block called the schedule parameter block. And at render time, it can call this block to change parameters in sample-accurate way.

So the first argument to do schedule is a sample time, the parameter value can ramp over time if the Audio Unit has advertised it as being rampable. For example, the Apple Mixer does this. And the last two parameters, of course, are function parameters -- are the address of the parameter to be changed and the new parameter value.

Now, things are a little bit different on the implementation side. We don't just get a pass-through call from the host.

Instead, the base class is going to fetch the internal render block, which we saw in the Filter Demo, and it's going to pass that render block the real-time events that pertain only to that render cycle. So the base class is maintaining the full schedule of all of the pending scheduled parameter changes and just parceling out the relevant pieces of it to the Audio Unit at rendering time.

So that's parameter scheduling, and we have done the exact same thing with MIDI events.

The host fetches a block from the Audio Unit before starting to render.

It calls that block at render time.

Now, you will notice here we have added a function argument called cable where, in the version 2 Audio Unit API there is only one MIDI cable with 16 channels, now we have 256 virtual MIDI cables. So if you have an Audio Unit that wants huge sample banks, you can do that. You can address them all on virtual MIDI cables.

On the implementation side for MIDI events, this is exactly the same as for scheduled parameters. The base class AU Audio Unit is maintaining internal schedule and passing events to the internal render block through the real-time event list only during the render cycle during which they are supposed to take effect.

So we think this is a big improvement. It saves the implementer a lot of work.

Now, about rendering in general, we are still using a pull model, meaning that an output unit pulls a mixer, pulls an effect, pulls another effect, pulls the player. Audio flows back down through the chain.

One difference here is in the version 2 API, the Audio Unit needs to maintain some state. It needs to have a notion of whether it's getting its input from another Audio Unit upstream or a function callback. Now, in the version 3 API, it's much simpler, the AU doesn't have to maintain that state. This callback, as we saw in the Filter Demo, comes from the host, and it is passed during every render cycle.

But otherwise, the APIs are pretty much functionally identical, and this lets us bridge very efficiently between them.

Now, if your host is calling AU Audio Unit directly to render as opposed to using AU graph or AVAudioEngine, here you will want to call allocate render resources as usual and then hold onto the render block.

You can call the render block to render.

It looks very similar to the internal render block.

It's worth reviewing here some rules about the audio buffer lists that appear at render time. Now, the host provides an output audio buffer list, and in that audio buffer list the M data pointer can be null.

The Audio Unit must replace this with an internally owned buffer and at the same time, the AU has to promise that that buffer will remain valid until the next render cycle. This is all, by the way, exactly the same as with version 2 Audio Units, I'm just reemphasizing it because it's important.

Now, in the render block, we have some rules, similar rules but not the same, about input buffers. The host provides that input block, the AU calls it for input, and when the AU calls that block for input, it has to supply valid audio buffer lists with non-null M data pointers to that block.

Now, the host is allowed to replace those pointers to memory that it owns and can promise to keep valid until the next render cycle or deallocate render resources, and all of this accomplishes an important goal, which is to absolutely minimize copying.

Okay. Here is the scary slide for those of you who are writing code to run in rendering context. So audio rendering almost always happens in a real-time thread context. And this is a restrictive environment because we can't allocate memory, which means that we really shouldn't even be calling dispatch async, for instance. And in fact we can't make any call at all which might block, for example, taking a mutex or waiting on a semaphore. The reason is if we do block and we block for any length of time, then the audio rendering thread in the system will miss its deadline, and the user will experience that as a glitch.

So we have to be very careful when both using and calling or, I'm sorry, both using and implementing these render blocks.

So you will see in the Filter Demo how we went to some lengths to not capture our Self object or any other Objective-C object for that matter.

In that block, we avoid the Objective-C runtime because it's inherently unsafe. It can take blocks. Unfortunately, the Swift run-time is exactly the same way. So this is why in the Filter Demo you'll see we have C++ objects and we capture pointers to those C++ objects. Now, if you are allergic to C++, you are free to do the same thing in plain vanilla C, although I'm not sure why you would want to do that. Enough scary stuff. I would like to bring up Alec Little now to show Audio Unit extensions in Apple Music creation apps. ALEC LITTLE: Thanks, Doug. I'm Alec. I work on the music creation applications for Apple, things like GarageBand and Logic. We are excited about the new Audio Unit extensions. We think they will add real power and creative possibilities for developers and users. So I wanted to talk a little bit about what some of our plans are. So first of all, we plan to support the Audio Unit extension, of course, in all of our main applications. So that's GarageBand iOS, GarageBand Mac, Logic Pro X, Logic Pro 10, and Mainstage.

So what I thought we would do today is look at some examples, some pretty pictures, if you will, from GarageBand iOS. And this is just preliminary stuff, but I think it will give you an idea of kind of what we are planning to do as a host to support Audio Unit extensions.

So first of all, we are going to be supporting AU instruments. So the example I'm going to be giving is about how we're going to implement those AU instruments.

So first of all, just a little graphic to explain about what we are going to do, something simple here, but GarageBand is going to request from the View Controller a custom UI dimension that we will talk about in a second. Pass MIDI events over to the Audio Unit and then of course receive audio back over the audio bus. On to the promised pictures.

So GarageBand, our main launch screen launches into what we call our touch instrument carousel. We have all of our touch instruments here: keyboards, drums, smart guitars, all of those things.

Then if you see on the left there is this container, and if GarageBand is seeing that there are Audio Unit instruments installed on the device, we will show that container. I can swipe over to that container, and that's where my Audio Units live. If I tap on it, then we will see all of the Audio Unit instruments installed on the device. Now, if I tap on one of those instruments, we will show a big gray box and a keyboard -- no. We will show your custom UI up there in that nice embedded view inside GarageBand, and I think that's the coolest part of this whole thing is that we get to show the actual identity of your Audio Unit inside our host there. And we will be providing our standard GarageBand keyboard for you to be able to play that. We will be recording the MIDI and receiving the audio back. So that just kind of brings up a pretty obvious point. When you are providing these custom UIs, make sure you are not putting any sort of custom, you know, MIDI controller-type device there because we won't be capturing your MIDI in GarageBand.

Just a quick look what we plan to do on the phone.

Is, again, we have a lot more limited screen real estate there, so there is a button in the top-right corner, which pulls up the controls view, and there is the custom UI. So all of the control there, and a little bit of room for the user to still play on the keyboard down at the bottom. So here is the most important slide probably from me today, and this is the dimensions we will be requesting from the View Controller.

So you want to have your stuff look good in GarageBand, pay attention to those dimensions, and we will do something pretty cool together. So, again, we think it's going to be really exciting to be able to see and have the users be able to play with the actual interface of your Audio Units right inside GarageBand, and we are really excited to work with you guys to come up with really cool stuff! DOUG WYATT: Thanks, Alec.

So I imagine you may have questions at this point. I would like to try to anticipate a few of them. What about inter-app audio on iOS? This is a couple of years old now.

And there is a number of apps that have supported it. Well, from our point of view, this uses a small subset of the version 2 API, and it doesn't support a bunch of thing that people have asked us for, like parameter support, presets, and so on. And we get these requests and I think, well, we should have a full plug-in model, which is what we have now. So while we are not deprecating inter-app audio, we do see the way forward to add all of these missing features is through Audio Unit extensions. Now on OS X, you may be wondering about the compatibility story if you have existing hosts and Audio Units. The bridges should save you from a lot of pain. They are compatible, and we have gone to a lot of work to make things work as well as we can. So I would recommend that you update to version 3 when you can or need to for some feature. Maybe you want to redo the way you handle MIDI events or scheduled parameters, for example.

We do have a shortcut for porting. It's called AU Audio Unit v2 Bridge. This is an AU Audio Unit subclass that is implemented on top of a version 2 AU, so you might be able to start with that and evolve to a more fully native implementation. And as Michael mentioned earlier, version 3 Audio Units are largely cross-platform between iOS and OS X. The signal processing code in AU Audio Unit should be absolutely fully portable because there are no UI implementations, or dependencies, rather. AU View Controller derives from either UI or NSViewController so you get a little bit of insulation, but you will at some point run into platform-specific UI. We are running out of time, and there is way more than I could possibly go through in an hour here. At this point I would like to refer you to your header files in the Audio Unit framework. You do need to link AudioToolbox for historical reasons. The main header file is AU Audio Unit.h but there are others, AU View Controllers in the Core Audio kit framework, and as we mentioned, there is the AVFoundation framework with AU Audio Unit component.h. We have good HeaderDoc in all of these, so I would like to urge you to check them out.

Finally, if you would like to use our Audio Unit's logo -- there is a white version, too -- you can go check out this link for a license.

And that brings us to the end. So we have seen how we now have for the first time a full plug-in model for audio on iOS, and it's the same on OS X. And, again, it lets you sell Audio Units in both the iOS and OS X App Stores by packaging your Audio Units as app extensions.

We looked at the simple host applications that are possible with AVAudioEngine, and that's illustrated in our new sample, Audio Unit v3 Example. So I would like to encourage you to write bugs as you work through the sample code, read the documentation, and work on the great AU hosts and implementations that I know you will. So for more information, this was our session yesterday, so thank you very much! [Applause]