The Life of a Button

Hello everyone, I'm Julian, later we'll be joined by Hugo who designs sounds across our products. Before we begin I wanted to explain a little bit about why we're going to be talking about a button for a while here.

I design and prototype user interfaces for future hardware. Because of the nature of my work I can't always use standard UIKit. I have built and rebuilt a lot of basic user interface elements like scrolling, zooming, paging, and of course buttons and I've learned a lot about the details involved in designing and building these controls.

So I'm here to share a little bit about what I've learned with you. I want to show you how something so simple, a button, has a lot of consideration and complexity behind it. And I hope that through looking at this button you can learn a few things.

I want you to use standard controls more thoughtfully.

To consider when to build your own custom controls. And when you do build custom controls I want you to consider your interactions all the way through.

So let's get started.

Hugo and I are expanding our artisanal toast app empire.

We are building a new simpler app that is connected to a toaster. If you imagine for a moment you're getting ready in the morning, you want to make sure you have breakfast ready by the time you're heading out the door. All you have to do is launch our app, press a button and expect a fresh piece of toast waiting for you in your kitchen. Now this works with our connected toaster. Our connected toaster goes through a few different states, it's ready and waiting for a new piece of toast, it's making toast, and finally the toast is done, the important part. But the way that our connected toaster ends up transitioning between these states is people using our app. When they use our app they can request a piece of toast to start the toaster going.

When the toast is done they will of course eat it. Maybe for some reason they'd want to cancel it, I'm not really sure about this one. And of course we want to make sure our toaster is always ready to serve. I'm not going to disclose exactly how this works but trust me it does and it's great. We also have big plans for the future, we could have AR toast, queues of toast, multiple toasters per household, and eventually we'll connect this back up to our toast social network.

My point is there's a lot going on in here and for today's session we're going to dive deep into the details of a single screen, in fact a single button on a single screen of our app. If we get to the details of our most important interaction right the rest will flow from it.

So if we look at our system again I think the most important interaction here is actually when we request toast. Obviously, all the rest of this has to work, but requesting toast is the first impression and if people aren't able to start the toaster the rest of it doesn't really matter.

So to do this today we're going to start by talking a little bit about what a button is, then I'm going to share with you a way of thinking about interactive controls. And finally, Hugo will join us at add some sound to our Button.

So what is a button anyway? The buttons that we are talking about today or onscreen user interface controls.

Buttons are indirect controllers of action.

What I mean by this is when you tap a button the action happens somewhere else. This is as opposed to a direct interaction where here I'm grabbing the lever and controlling the lever directly.

Direct interactions can reference the real-world experience and feel more fun. But the real power of indirect interactions like buttons is that they can be clearer and more powerful because of their separation.

The interesting thing here being that even the physical buttons that these onscreen buttons are imitating are indirect.

Even though the action, the result is happening somewhere else the button controls that action. And just like the action a result can be designed a button can be designed, we can consider both of these things separately and connected.

To do this today we're going to be relying heavily on a single lens of design feedback.

Feedback is how you let people know what your app is doing.

I'm sure a lot of you have thought of a variety of different ways that you can provide feedback within your apps and are already doing so. But I think there are two major categories of feedback.

The first is that you can tell people, you can literally put text on screen explaining what's going on or design icons that explain things.

The second is that you can show people, you can allow people to experience the results of their action more like the real world through visual, audio or even haptic change over time.

Now we're going to apply feedback to three phases of an interaction, not just its completion. We can think of these phases in terms of a physical button.

Before I have pressed the button it's just hanging out there. During my interaction with that button it's being pressed in. And finally after I've pressed that button my finger is lifting off of it and it's returning to its original state. So what does it even mean to be providing feedback before I have begun interacting with a button, why does this even matter, the button's just there right.

Well if we think about our iPhones for a moment, the iPhone screen is just a piece of glass.

This piece of glass happens to have glowing pixels underneath it and what matters is what people think those pixels mean.

Glass doesn't tell you that it's tappable, but something on screen might. You could see that you could page through something, drag something or of course for the purposes of this presentation tap on something. The academic term for what I'm talking about is perceived affordance. Affordance refers to the connection between an object and the person interacting with it. A person might know that they can grab a handle or press in on a physical button. But the perceived part of this is talking about what people know about those glowing pixels on screen. People only know what they can interact with on that iPhone screen because of their prior experience having used iOS before and their current context, the fact that they're on this phone in an app that they may or may not understand.

So let's apply this to our toast app of course. How do I know what will happen when I tap this button, I just put a button up and call it okay right, no. What we do is provide a label, a label is feedback in which you are telling people literally what is going to happen. In this particular case I tried the label toast, but unfortunately toast refers to both the verb to toast and it also refers to the delicious piece of bread you're going to be eating for breakfast. So what should we call this label, toast toast? No. We could try to design an icon for this button but I think this isn't particularly clear, especially for something that people will be encountering for the first time in an unfamiliar context. So we thought about it a little bit and we came up with the phrase make toast. I know that in our diagram we refer to this as request toast but we spent some time thinking about what people using our app might understand and what feels most comfortable for our app.

The next thing to think about is how do I even know that this is a button, just a rectangle floating on a black background might not be enough. But if you add some context like an iPhone screen you might start to understand it or you could add some prior experience. It could be the shape of the button, the manner in which is rendered or if it's used consistently enough just the color of text. In this case, I don't think people are going to be very familiar with our app they're going to be launching it once a day, more if we're lucky so we're going to focus on more standard iOS button shapes because we don't want to throw in something new. So we try these out on screen.

Another thing that we can think about with respect to feedback before we begin using this button is how do I even know what this button relates to. It could be something about where in the app a button is located, these buttons have different meaning and feel. Or it could be something about how it's grouped with other controls or how close it is to other objects on screen. I'm starting to like this last one here so we're going to try it out on a real screen. I drew this user interface in Keynote and I put it on a phone and I'm holding this phone in my bedroom where I might expect to use it in the morning before I order toast. And I'm just trying to understand do I know what this button will do.

The next thing we should think about is what happens during my interaction with this button. For a physical button I'd be pressing it in.

Now the term feedback refers to telling people something about the result of what has happened and in this case, we haven't actually done anything yet, so we might use the term feedforward for this.

Because feedforward refers to helping people understand what is happening while they're interacting with this button.

Feedforward is a component of making interactions feel fluid, you can convey responsiveness for things like 3-D touch, pinch or slide movements.

Thinking about how feedforward applies to our button we should be thinking about what is happening now, what is happening while my finger is making contact with the screen, with that button. In the case of our overall toast system we are in the middle of touching a button to request toast. So the part we're going to focus on is the toaster being ready and eventually transitioning to the toaster making toast. We should be thinking about what is happening and how I can hint toward what will happen.

Let's say I've begun touching this button, how do I know that I've begun tapping this button, why does it even matter? Well, for one, the hit area of this button might not be what you expect, if you look closely at the circle maybe I'm not quite touching it. But perhaps the hit area is actually a little bit larger than the visual size, this is particularly important for very small controls if we made that lever interactive.

So how do I know I've begun tapping this button? Well you might be tempted to animate the button, to have it fade in or grow very slowly, but this can make things feel unresponsive.

This kind of animation might be appropriate for 3-D touch if it's responding continuously as I'm pressing in. But for a quick contact with the screen we might want to do something faster, if we are going to animate it do it very quickly. So what we do is provide a confirmation sound, a haptic or visual change. In this case, we're just going to highlight it instantaneously, make it feel a little bit faster. This also helps us know that a button is enabled and that it can work.

Maybe we try connecting the feedforward on the button to the feedforward of the toaster itself, as I said we can start to hint toward what will happen. So here what I'm trying is as I touch this button maybe we show the piece of toast starting to appear. I feel like that might be a little bit too much for this button though, so I'm not going to go forward with that but I tried it. Another thing that we should think about with feedforward is what happens if I change my mind, let's say I'm kind of groggy in the morning, I pick up my phone I accidentally touch this button but I don't want to start my toaster yet. Well for most buttons I can actually drag my finger outside of the button to begin canceling it.

This is something that's actually very important for all sorts of fluid interactions because it's what makes a fluid interaction undecided. If you're paging through different screens you don't know which screen it's going to end up on until you lift your finger and the same thing applies to a very simple button. I can bring my finger back inside the button and have the same feedforward show me that I am now activating this button again. Now that we've decided a little bit about the feedforward that we're going to have we should try this out on a device as well. So again, I know this seems really simple for a button, but this is very important for scroll, pinch, paging, a lot of touch interactions.

Finally, we should think about how to apply feedback after I have lifted my finger up off of this button. For a physical button it's going to return to its original state. For our onscreen button we're going to have lifted our finger off the screen.

This is of course the realm of classic feedback that most of you are familiar with. But the thing I want to point out here is that we can be thinking about feedback both for the button itself and the action, the result that is happening, and we can think about how to connect the two so people understand what has happened.

Again, within our system we're only focusing here on getting people into making toast.

So what happens when I lift my finger from this button? Does the unhighlight happen immediately or is there a slight delay so that even fast taps can be seen? Is there a timer, does this button support double tap, if so I may have to wait a little bit before showing the confirmation? If we think back to the two types of feedback we can just literally tell people what is happening. I can put some text on screen, I think it's very clear but it also requires reading and more text.

So maybe we can show people, maybe the button itself provides some feedback here it's flashing or maybe it's something that represents the action that provides feedback. For example, the icon of the toaster itself could show you that it's starting to toast. Or what we can do is we can just do both, I can put text on screen and I can show the toaster. This is great because I might have missed that animation, but I can still see what state the toaster is in.

But similar to interacting with the button with feedforward I should think about what happens if I change my mind. I've started toasting but maybe I want to cancel, maybe I want to stop the toaster. We could replace that button with a stop toasting button, but I think that's not particularly clear because this button looks an awful lot like the make toast button that was up there before.

So maybe there's a separate stop button. I feel like that red is a little bit too much, so maybe we try that same button but separate from our make toast button. This is starting to feel a bit better, so what I should do is try it on device again. We should see how the feedback on the button itself connects to the feedback from the toaster communicating what has happened.

To recap a little bit of what I went over.

Feedback is how you let people know what it is that your app is doing. You can literally tell people, you can put text on screen it's fine or you can show people, you can allow them to experience the change over time. We can apply feedback to all three phases of an interaction, even a simple button. And something I was implicitly doing here is trying out options to understand what's too little and too much. The only way I can really know if an animation is appropriate or not is to try it.

So what we've done here is we've designed a very important part of a much larger system, but I think it'll be a lot easier to figure out the flow of the rest of this once we get this one piece right. Now that our button is designed and pretty great I think we can start to think about sound, so I'd like to invite Hugo up to the stage. Thank you Julian. Hello everyone.

Now we can make this quick and add a simple click sound to this button. Sounds pretty good or could it be better? Could Julian have done a better job and invited me a little earlier on in the process? I'd like to tell you a few things about how as I a sound designer could approach a project like this. I'll talk about where to look for inspiration, about designing the sound, and some key building blocks that we use while doing this.

But before we dive in why do we talk about sound? Now some of you may think our users mute the sound on their phone quite often, is it important to add sound to my app? Well of course it's up to you but George Lucas has said that 50% of the experience of a movie is determined by what we hear, by its music and sound design. And sound is all around us, we use it to navigate through the world.

And sound can help enrich the experience of your products. And it helps us remember the experiences of your app, as well as your brand, so it can have a big impact.

So where do we look for inspiration when we want to design sound for a button? Well we can start by taking a little detour and looking at some real-world buttons.

Some of them are made of cheap materials and sound like this, while others are made of high quality materials and sound like this. Well this was actually a recording of the volume button on an iPhone X, of course we amplified it quite a bit.

These sounds though are essentially byproducts, they're not necessarily designed they're defined by the materials these buttons are made of.

But because we're designed for software we have a lot of freedom, so we can choose to not play any sounds at all and often that is what we do. But if we do decide it's useful to add sounds to UI elements then we can take cues from real-world analogs to add meaning to these UI sounds and to help our users understand them. Now do you want to add sound to your app? Well it depends, think about category your app falls in, who will use the app and what will their expectations be, and what is the context, where will the app be used. For our app we know it will be used by people who like to eat toast and they're used to making toast in the kitchen, but from now on they can make it from anywhere in the house. They can still though use the audible cues they're used to getting from their toaster in the kitchen, so for that reason we think that this app is a perfect candidate for the use of sound. So let's look at the process of making toast and listen to sounds we encounter.

Let's define if these sounds are useful and if they are if we want to use them in our design. So we'll use Julian's timeline for this and listen to the sounds that this toaster makes.

First, we pull down the lever and then we press it in place.

Now what we hear then is the toaster heating up with a subtle buzzing sound of the coil. Until our toast is done and the toaster ejects the toast. Now let's go through these steps one by one.

First, when we pull down the lever we feel the resistance of the spring while we push it down, it's almost like if we compare it to what happens in our app it's like designing sound haptics and animation and synchronizing them all. But in our app, we decided to replace this with a simple button so we don't need to design a sound for it. And when we request the toast that is when we press our button, the next step, and that is what Julian asked me to design a sound for so let's get to work on that. We can record sounds, we can use electronic instruments, synthesizers to generate sounds and then we can use software to scope these sounds into the right shape. Now I'm not going into great detail about this today, but please watch last year's talk Designing Sound if you want to learn more about this process. In the end, we came up with three different options.

This is option A, this is option B, and this is option C. Okay now option A is quite minimalist, it's so simple, it works pretty well with the simple UI of the app but it feels like there's something missing. And we found out that while we were designing these that it's much more satisfying if these sounds have two clicks, one when you press down and one when you lift off your finger.

Now option B has this character, but and now it is a sound that's much more realistic, it doesn't really sound like a toaster but it does have this physical quality to it. For the same reason though it does sound a little harsh and a little metallic. And then there's option C, it's a bit of an odd one but it does have character and it's a sound that I would recognize if I hear it again if I use my app again. Another thing that's good about this sound is that it has this tonal confirmation built in, it's like throwing a checkmark telling you your action succeeded. I'll play it one more time see if you can hear this. And for all these reasons I think that option C is the one I would recommend going for, it's a simple sound, it has a friendly character, and it's something unique it's not something I'd recognize from any other app. Okay so it looks like we're done, at least we made our button click but we're not done making toast yet. So let's keep going, maybe there are some other opportunities for the use of sounds in this app. Now as good sound designers we want to tell the whole story.

So after pressing our button the toaster heats up and we hear this subtle buzzing sound. Now we can -- is this helpful at all this sound? Maybe, it tells us something about the state of the device. So we can choose to play a sound that resembles this on our app as well, maybe a continuous sound, something like this. But making toast may take several minutes and having this sound play over and over again might not be the experience we're looking for right. So how about something a little more subtle like a timer sound that fades in and out every half a minute or so.

That could work. But does this really need a sound at all because what we're really waiting for is what comes next, that's when our toast is done and this is what we want to notify our users about, that's the important moment. So we want to play a clean and simple notification sound that they won't miss, something like this. Okay so now let's listen to these sounds together. First, we press our button and then we decided for the next step when the process is active we don't really need a sound until we get our notification and the toast is done. And it's important to listen to these sounds together to make sure they family well together because this is how our app communicates to its users, this is the voice of our app.

So now our app is in good shape. I'd like to tell you about some key building blocks of sound that we use when designing any kind of sound. The first one is timbre or tone color. And most of you will instantly recognize this one. It's a piano right, but how do we know it's a piano? How do we know it's not a vibraphone? It's determined by the material an instrument is made of, by its shape, and how it's being played using a mallet, a hammer, a bow or your hands. So these instruments can play exactly the same tone but sound very different. Now for our app we decided to go for a character, for a timbre that is friendly and it's not too harsh or metallic.

The next one is frequency or the pitch of the sound. Some instruments play at very high frequencies and others play at very low frequencies.

But also nonmusical sounds can be played at very high or low frequencies. And when we do this it gives an indication of the size of an object. So if we take our toaster and we record the toast eject sound, we play it at a very high frequency it sounds like a tiny little toaster.

But if we take the same sound and play it at a low frequency it sounds like this big giant toaster.

So next up is duration or the length of a sound.

Sounds can be very short or ring out much longer. And if we design UI sound and we know our button is going to be pressed a lot of times in a row it's important for this sound to be very short and subtle.

But if it's only pressed once in a while, which is actually the case for our toast app, it's okay if it's a little longer, if it's a little more elaborate.

In fact, this sound only plays once every time we use our app.

And then there's loudness or amplitude or volume.

Now when we design ringtones or alarm sounds we don't want our users to miss them, we don't want them to miss their phone calls or sleep through their alarms. Our audio engineer Mitch is calling me from the back of the room and I hear it loud and clear. But if a UI sound would be loud like this it would become unpleasant. It only needs add this subtle layer to the interaction. Now let's quickly reply to Mitch. Now if I would miss this keyboard sound because I'm in a noisy environment it's not a real big problem, but I definitely don't want to miss my ringtones.

So to recap the four building blocks of sound, our timbre, frequency, duration and loudness. And when you start thinking about designing sounds and adding sounds to your app it's really good to keep these in mind. And that concludes what I want to tell you about sound today, but before we end the session these are the most important things that Julian and I want to leave you with. Details are designed even if they seem obvious, even if they're as simple as a button, even if they're as simple as a click sound.

Now you can take inspiration from real-world elements, but there's no need to copy them one-to-one. Also use the freedom and the flexibility of designing for software to create something that is completely unique and fresh. And then what we see, the animation, and what we feel, the haptics, and what we hear, the sound, they all combine into this one single experience.

And lastly, but most importantly, learn by trying things out.

You know we tried out a lot of things today and some of them didn't work that well while others worked great. But the only way for us to find out was to try them out first.

Now I hope this all inspired you to spend some more time on the details you might normally take for granted.

If you get these details right it can make your apps even more amazing.

Thank you for listening, have a great day. [ Applause ]