Introducing Parameters for Shortcuts

Hi everybody. My name's Ari and I'm really excited to be here to tell you today about some new updates to Shortcuts in iOS 13.

In iOS 12 we extended SiriKit by introducing the Shortcuts APIs. Which let you expose actions from your apps to Siri.

Shortcut actions exposed from your apps can be used in three different ways. They can be used as suggestions. They can be used through voice. And they can be used as part of multi-step shortcuts. Now, with suggestions, Siri will suggest your Shortcuts to the user as just the right time in search and on the lock screen to accelerate things they do with your apps every day. With voice, customers can use any capability of your apps just by asking Siri.

This is really powerful, because Shortcuts makes it super easy to voice enable your app, which lets people use your app in totally new ways. Whether they're on the go with AirPods, in the car with CarPlay or at home with HomePod, or just from across the room with "Hey Siri". Lastly, users can build their own shortcuts in the Shortcuts app, which include multiple actions, including those from your apps. Now, this year we're talking Shortcuts APIs even further and we have some really exciting API updates to share that take voice and multi-step shortcuts to the next level. So let's dive into what's new.

In iOS 13 the Shortcuts app is redesigned, and it's built into iOS, so it's easier than ever for people to discover and use Shortcuts.

Now, after we announced Shortcuts last year, the biggest request we heard from developers was to add parameters. So we're really excited to share that in iOS 13 we're introducing support for parameters and it enabled something we call conversational shortcuts. This basically allows users to have a conversation with your app and you control the conversation. So you can prompt the user for information you need and respond dynamically based on the users input and their current context. This opens up really totally new use cases for what people can do with Siri and apps, and I'm excited to tell you more about it in just a second. But next, we're adding new ways for users to customize your shortcut actions through a new customization UI that acts on the same parameters as the ones that you define for the voice interaction. You can also enable advanced scenarios like outputs and variables. Lastly, we have an updated Add to Siri UI, which makes it easy for people to get started with Shortcuts inside of your app. So, let's take a look at what some of these new capabilities mean to the voice experience inside of Siri.

I have a great new Siri shortcut. It's called Add to Instacart, and it adds items to my cart. So, with conversational shortcuts, when I say Add to Instacart, Instacart can ask what I want to add with custom dialogue. And I'll say bananas. So, there are a few different options for bananas. Instacart will ask which specific one I want by showing a disambiguation. I'll stay I want the organic bananas. So, if it's contextually appropriate, the app can ask additional questions based on what I've said so far. In this case, Instacart will ask how many bananas I want, and I'll say six. Lastly, Siri can let me know that the task is complete with custom dialogue and custom visuals from Instacart.

I also have another new shortcut. It's from the MLB At Bat app, and it means that I can just say, "Hey Siri, watch baseball." and it will jump me right into my favorite team's games.

So, the Philly's aren't actually playing right now, but with conversational shortcuts the MLB app can detect that and recover appropriately. Letting me know that there isn't a game on right now but giving me some other options of things I might want to watch instead. So, that's just a glimpse of what you can do in Siri, and in a minute we'll dive into the implementation. But first, let's talk about some updates to the Shortcuts app in iOS 13.

The Shortcuts app is now built in to iOS. And you don't need to download it from the App Store. It's the one place where users can manage all of their shortcuts. So everything that used to be in settings in now here also. In the My Shortcuts tab the user can see all of the shortcuts they have set up and they can run them just by tapping on them or by saying the name to Siri. And there's no need to record a phrase first. In iOS 13 we also have a brand-new Automation tab where people can set up shortcuts that show up or even run automatically based on certain criteria like when they arrive at a particular location, when they get in the car, or when their alarm goes off in the morning. So, we all know Shortcuts is really useful, but it can be hard to start from a blank slate. So users can find hundreds of pre-built shortcuts to get started with in the Gallery tab. And this year, we're updating the gallery with a new section called Shortcuts from Your Apps. This section highlights relevant actions from third-party apps which includes your apps. So this is a really great opportunity to get your apps actions in front of the user even when they're not using your app. The shortcut editor has been updated also with a new natural language format. Here's what it looks like in the editor when the user sets up a shortcut with the Soup Chef app. Note how my app has exposed this description, Order 1 Clam Chowder. And the user can tweak those parameters at any time just by tapping on one of the buttons. Like tapping on 1 or Clam Chowder.

The editor also has a new action pane which features actions from third-party apps that you can drag or tap to add to your shortcut. Lastly, your apps actions can now provide outputs, which means they can be hooked up to other actions to create even more powerful shortcuts. For example, the Soup Chef app can share an ETA. So you can add it to an Add New Reminder actions to make a shortcut that adds a reminder to pick up your soup exactly when it's ready. So, that's what's new in the Shortcuts app. But we also have some updates to Add to Siri.

Now, traditionally, voice functionality can be hard for users to discover, because they don't know which features of your app are available or what to say in order to use them. Add to Siri makes this easy because people are already using your app. So you can add the Add to Siri button in the relevant parts of your app and it -- the user actually wants to use it with their voice. In iOS 13 we've redesigned the Add to Siri sheet. The new UI lets the user set up shortcuts without having to record a phrase. They can just type a phrase or they can accept one that's suggested by you.

Afterwards, saying the phrase to Siri will run the shortcut. And if you tap on the action under Do, the user can tweak the behavior of the shortcut just like they can from inside the Shortcuts app. Now, let's dive into the implementation. First, we're going to talk about how to enable your shortcuts to be customized by the user and then we'll cover parameter resolution and how to use it to implement a great voice experience. Then, we'll talk about how to deal with parameters that have dependencies on each other. And lastly, we'll cover how dynamic options and outputs can help take your actions to the next level in the shortcut editor.

So, let's start by enabling Shortcut Customization for an app that I've been developing.

I'm really, really into soup and so I've been working on an app called Soup Chef, which lets people order soups from their favorite restaurants. I want to enable shortcuts that let the user order a soup and choose how many soups they want right from inside of the action.

So, in order to set this up let's start in the Intent editor in Xcode.

As you know, an Intent is a specification of a task that the user can complete in your app. In this case, ordering soup.

You should define parameters for every piece of information that you need from the user in order to complete that task. The set of parameters that you define here is used for both the set of questions that Siri will ask in conversational shortcuts as well as the set of parameters that you can edit in the shortcuts editor.

From each parameter that you add to your Intent, you can choose from the list of supported types that are provided by the system or you can create an Enum, or you can even in iOS 13 create your own custom types. Once you pick a type, you'll see some additional settings that let you customize that parameter which are specific to that type. For example, if I pick String, I'll see some new options for capitalization and autocorrect behavior.

For this parameter I'm going to choose a soup enum type that I already created. Now, since I'm updating an existing parameter, I need to check the User Facing checkbox so that this parameter can be exposed to Siri and the Shortcuts app. For new parameters that I add, this will be checked by default. Next, I'll need to fill in some new information for my parameter. I'm going to add this Display Name, because that's used to show the parameter inside of the shortcut editor. Then, because this is an existing Intent, I also need to check the Intent is user-configurable checkbox, which makes the Intent as a whole available to both Siri and the Shortcuts app. Now, lastly, I need to fill in a parameter summary. I want to include the parameters quantity and soup in the parameter summary so that they're really easy for the user to tweak in the shortcut editor. So, I'll add a string, order quantity soup, and I'll let Xcode turn quantity and soup into variables for me in Xcode. So that's it. That's actually all you have to do to get your shortcuts to be exposed like this in the shortcut editor. Now that the Intent is configurable, and it shows up in the Shortcuts app, it's ready-to-use parameters. So, the next thing we need to be able to do is resolve the parameters of the Intent. Let's talk about how to do that. In iOS 12, when the user invokes a shortcut through Siri it goes through two phases. First Confirm and then Handle.

This year we're adding a third phase which is called Resolve. And it should be familiar to you if you've already worked with SiriKit APIs in the past.

In the Resolve phase Siri goes through each parameter that you've defined on your Intent and asks your Intent handler to resolve that parameter. To decide whether or not to ask the user a question. For Soup Chef, Siri will first call Resolve on the soup parameter which might ask a question like, "What soup do you want?" And then it will call Resolve on quantity, which could ask a question like, "How many soups do you want?" And lastly, it will ask it on store location, which might say, "Where do you want to order your soup from?" Once all the parameters are resolved, Siri will invoke the same Confirm and Handle methods as the ones that we had in iOS 12. So, let's take a look at what this means in code. Because Xcode automatically generates an Intent handling protocol for every Intent that you define. And you implement that protocol in your Intent handler. In iOS 12 Xcode generated these two methods; Confirm and Handle, which you would implement in order to execute your Intent. But this year, Xcode will also generate Resolve methods for each parameter that you've marked as configurable. Siri will call the resolve methods and ask questions in the order of the parameters that you've defined in Xcode. And you can drag and drop the parameters in Xcode to change the order of parameter resolution.

When the Resolve method is called, the Intent is passed as input to the Resolve method, including a value for the parameter that's currently being resolved. So that value can come from a couple of different places.

The first time the Resolve method is called, when the shortcut is being run, the value is filled in from whatever the user set on the shortcut when they created it. In this example, the soup field is empty, maybe because the user sometimes orders clam chowder and sometimes orders tomato soup. And they want to set up a shortcut that's going to ask them every time they use it which one they want.

So, when the Resolve method is called the soup parameter will be filled in as empty because there was nothing filled out in the shortcut and that's going to be passed to the Intent that's passed to Resolve. The responsibility of the Resolve method is to provide a Resolution Result which determines what Siri will do next. Now again, because the field is empty, the Soup Chef app might want to ask the user for a value by passing the Needs Value Resolution Result.

Passing Needs Value will cause Siri to ask a question for this parameter which the user can respond to with their voice. So Soup Chef might say, "What type of soup do you want?" And the user can respond with something like clam chowder. Now, the input that the user says, clam chowder, will be filled in to the Intent and the same Resolve method will be invoked again with that input this time. So the app can decide what to do next. In this case, clam chowder is a valid soup and we have it in stock. So the Resolve method should this time return success which will cause Siri to move on to the next parameter in the list. Now, if the user had set up the shortcut instead to have the parameter preset to clam chowder, because maybe they want their shortcut to order clam chowder every time without asking them, then that would be passed to Resolve instead of the empty value and in this case, the Resolve method can return success right away and Siri won't ask a question for that parameter at all. So, in the implementation of your Resolve method, you need to look at the current value of the parameter on the Intent. If there's no value, you might need to ask the user for a value by returning Needs Value. And if the user has just -- has filled out a value on the shortcut when they set it up or if they just said something to Siri, there might be a value there already which needs to be validated and hopefully passed as success. So every time your Resolve method is called, it needs to supply this Resolution Result. And I'm now going to hand it over to Roman to tell you about how you can provide great Resolution Results to provide robust voice experience in Siri. Roman. Thanks Ari. Hello. When I'm building a voice experience I want to think about all of which cases that the user could say and how to handle them well. For example, if the user asks to order more soups than we have in stock we should tell them appropriately. If the user asks to order soup from a store that's far away, we can give them options of stores that are closer. On the Resolve step, this is your opportunity to influence Siri's behavior and decide whether or not Siri asks a question. And this is where you handle all which cases of the user input.

In order to do that, you need to provide the Resolution Result.

There are six Resolution Result types that you can choose from. Let's go through each one of them starting with needsValue.

If the user didn't specify the value at the shortcut configuration time and you need one in order to proceed, you can tell Siri that you need a value.

You need to provide your own custom prompt dialogue for each parameter that you can resolve. To do that, open the Intent editor. Select the parameter that you would like to specify Siri dialogue for and then type your custom prompt in the prompt text field.

If you return disambiguation Siri will ask the user to pick from a list of values. This is a good idea if the value in the parameter is ambiguous or if you just have few possible things for the user to choose from.

You can customize the disambiguation dialogue in the Intent editor. Return unsupported tells Siri that your app doesn't support the provided value.

For example, if customer tries to order too many soups we can respond appropriately by saying that we don't have enough in stock. And after that, Siri will re-prompt the user. We can define the error messages that can be shown to the user under the Validation Error section in Xcode.

Xcode automatically providers default behavior for medium and maximum values that you specify in the Intent editor. Return confirmationRequired will cause Siri to ask the user to confirm the parameter value. Use this if it's unclear whether or not the user actually wants this value or you have a strong guess, but still would like to confirm the value with the user.

Again, you can customize the Parameter Confirmation prompt in the Intent editor. Return success means that we have a valid parameter value and Siri should move on to the next parameter. Return notRequired means that your app doesn't need a value for this parameter at this time, so Siri should skip it and move on to the next one.

The Result methods will be invoked at the shortcut execution time in both Siri and the Shortcuts app. So you need to make sure you implementation behaves correctly in both environments.

So now let's take a look at what the execution flow looks like. This is the new Shortcuts app and here in my Shortcuts app you can see all of the shortcuts that I created or added from the Gallery. To create a new shortcut, I'm going to tap on the Create Shortcut button. In the shortcut editor I need to add actions to my shortcut. To do that, I'm going to tap on the Add Action button. I use the Soup Chef daily to order delicious soups, so here's already a suggestion for me to order soup.

All I need to do is just give it a name.

So here's my shortcut. All I need to do is just give it a name and save it to My Shortcuts. To do that, I'm going to tap on the Next button. Let's call this shortcut Order Soup.

Now let's try invoking the shortcut with voice and Siri. "Hey Siri, order soup." Which soup do you want? We have three specials today. Clam chowder, tomato soup, or chicken noodle soup. Clam chowder. There are two restaurants near you. Which one do you want to order from? 38801 S Sheridan Blvd or 7401 S Lewis Ave? The first one. Ready to order? Yes, please. Okay, ordering. Your total is $3.75. Your clam chowder order will be ready in 10 minutes. And just like that, I ordered soup with Shortcuts and Siri. Now, let's try writing the same shortcut from the Shortcuts app. But before I do that, I would like to customize the shortcut.

I'm going to tap on the soup field and select the soup that I usually order from Soup Chef. I'm going to select clam chowder.

Now I'm going to tap on the Done button to save the shortcut.

Let's try writing the shortcut from the Shortcuts app by simply tapping on it. So here I get presented with the same disambiguation prompt to choose a store, just like in Siri. I'm going to select the first option.

And done. I just ordered another soup. I guess I'm really hungry today.

Now, let's talk about a few ways to take our shortcuts even further.

Wouldn't it be great if I never have to leave my home to have some soup? Yes. And that's why I'm adding something new in soup shop delivery. In the shortcut editor, I want the user to be able to choose between delivery and pickup options. If they choose delivery, let's just show them a field that lets them decide where it should be delivered to. For example, to their current location.

If they choose pickup, we should show them a field that lets them decide which store to order from. To make that work, we need to express parameter relationships in the Intent editor. So let's start by taking a look at all of the parameters involved in this. We have the orderType parameter which has two possible values; delivery and pickup. We also have parameters for deliverLocation and storeLocation. Now, let's take a look at the Relationship section.

Here for my deliverLocation parameter I'd like to specify the Parent Parameter to be the orderType parameter. I only want to show my deliveryLocation parameter if the parent parameter has the value of delivery. And exactly the same thing for the storeLocation parameter. I only want to show it if the parent orderType parameter has the value of pickup.

So now my users are able to easily switch between different order types.

Let's see how to do this in Xcode.

So I have the Soup Chef project open here. I'm going to select the Intent definition file. This is where I define all my Intents.

I already have Intent for order soup. Let's take a look at the parameters. We have soup, quantity, toppings, store location. Now let's scroll down to a new section in Xcode that we have called Shortcuts app.

Here you can see the -- all the Support Parameter combinations by the Shortcuts app and the corresponding summaries. You can also see the preview of what it's going to look like in the app.

So currently we have the storeLocation and we have the summary which says, order quantity soup for pickup from storeLocation. I didn't put toppings in my summary so it appears under More Options.

Now, let's go up and add a couple more parameters to support delivery. I'm going to click on the plus (+) button and add a new parameter called orderType. I'm going to specify a display name for this parameter.

So if I scroll down to the Shortcuts app section you can see that the Order Type parameter now appears under More Options and it was automatically added to the Support Parameter combination. Let's scroll up. Let's specify a new type for the Order Type parameter. I'm going to add a new enum.

Let's call this enum Order Type.

I will also specify the display name for this type. Now we add to add two cases; delivery and pickup. Let's start with delivery. I'm going to click on a plus (+) button.

Type delivery. I will also specify display name. Let's add another case for pickup.

Also provide display name. Since my users can refer to pickup as take out when they speak to Siri, we can add this as a speakable match. So I'm going to click on a plus (+) button and add take out.

Now let's go back to the Intent. Now we need to add another parameter called Delivery Location.

Let's also give it a display name, Delivery Location.

So now let's scroll down and see what we have there. We still have the single parameter combination which contains all of the parameters.

But it doesn't make sense to have both Store Location and Delivery Location at the same time. Let's fix this.

We're going to scroll up to the parameters section.

Select the Store Location parameter, expand the Relationship section, and select the parent parameter to be the orderType. I only want to show my Store Location parameter if the parent orderType parameter has the value of pick up.

Let's do exactly the same thing for the Delivery Location parameter. Let's select the parent parameter to be the orderType. Show if parent has exact value of delivery.

Now, let's go down to the Shortcuts app section and take a look what we have there. Xcode automatically created two more parameter combinations for us. Let's fill out summaries for each of them. So will be the Order quantity, soup for orderType. Now let's imagine user taps on the orderType field in the Shortcuts app and they select pickup. In this case, they will see the parameter combination that contains storeLocation.

So I already have the summary for this parameter combination, but I had a hard coded value for the pickup. Let's change this to be a parameter orderType.

And finally, we need to update the last parameter combination, which includes the deliveryLocation parameter. Let's add the summary. Order quantity soup for orderType to deliveryLocation.

Let's take a step back and take a look at what we just did. First, we identified parent and child parameters in our Intent. And then, we established parameter relationship between parent and child parameters. And finally, we updated summaries for each parameter combination so they look great in the Shortcuts app.

Now, let's talk about Dynamic Options.

Most parameters such as String or Number have obvious input mechanisms, an input field in the Shortcuts app.

We're asking a user to specify some values here.

For some parameters you may want to allow the user to pick from a fixed set of values and have those values be dynamically provided by your app when the user is setting up the shortcut.

For my Store Location parameter I wanted the user to be able to choose from a list of stores close to them.

In order to accomplish that I'm going to check the Dynamic Options checkbox for my Store Location parameter.

Once I do that, Xcode will cogenerate two additional intent handling protocol methods. The first one provides store location options. And another one for providing a default value for this parameter.

In your provide options method you need to return the current set of possible values I the completion handler.

You will also have the opportunity to provide default values for parameters with Dynamic Options enabled.

When your parameter of Dynamic Options is being resolved during the shortcut execution time and your return needs value in the completion handler of your Resolve method, Siri will automatically use the list of options you provided in the form of a disambiguation prompt.

Lastly, I want users to be able to make more shortcuts work together with other actions.

For example, a user could set up a shortcut that places an order and then sends a message with order details to their friend who usually picks up their order.

In Xcode 11 we are introducing the ability to define your own custom tabs that you can use in both Intent and Intent responses. So I'm going to define a new custom type called Order Details, which contains proper sets of information that I want to pass on to another action.

In this example I'm going to define two properties; Order Estimated Time and Total.

As you might know an in turn response is a logical result of an execution of an attempt. It contains properties that convey the result to the user.

To add an output, I need to define a new property of my custom orderDetails type.

And then, I need to designate this property as the output.

When the user taps on the Order Details variable in the shortcut editor, they will get presented with an option to choose one of the properties that I just defined in my custom type.

It can also use custom members of your custom types in response templates for Siri. So the same data that is passed as the output can be spoken aloud.

Today you learned how you can allow users to customize your shortcuts in the new built-in Shortcuts app. You also learned how you can provide conversational shortcuts in Siri. Choose the right parameters for each Intent so your app can correctly respond to the user each step of the way.

You can extend the power of your shortcuts with other apps using Outputs in the Shortcuts app. We have another session on Friday where my colleagues will talk about design and building great shortcuts. You can also find us in the labs throughout the week. Thank you so much for coming and enjoy the rest of your conference. [ Applause ]