Developing CarPlay Systems, Part 1

Hello, and welcome to our first session on developing CarPlay Systems. My name is Emily, and I'm responsible for Car Experience engineering at Apple.

CarPlay was designed from the ground up as a smarter, safer way to use your iPhone in the car. CarPlay takes the things you want to do with your iPhone while driving, and puts them right on your car's built-in display. CarPlay brings the familiar elements and interaction patterns of iPhone into your car. Users already know how to do things in CarPlay because they know how to use their iPhone.

CarPlay is about more than just showing iOS on the car's display. CarPlay is thoughtfully integrated with your car's built-in system to deliver the best possible user experience.

CarPlay allows you to design for the future. With CarPlay, you can build a car today, and when Apple introduces new versions of iOS, users will immediately benefit from new features. Your car will remain fresh for years into the future. Today we'll give you an overview of how the CarPlay feature works, and discuss the most important requirements for you to consider. We'll explain the user experience that we want to enable with CarPlay and the design guidelines that we've provided. At the end, we'll talk about CarPlay supported apps. IPhone user interface is designed to work perfectly when held in the hand, when you can touch the phone.

CarPlay translates that user interface to the car, making it appropriate in that setting, and safer to use.

It works with what you already have in your vehicle. It's comfortable to use in that environment.

Now let's talk about how CarPlay works in detail.

Let's start with the native user interface of the vehicle.

When an iPhone is plugged in, the CarPlay session begins. When a CarPlay session is brought up, information is sent both directions via communication protocols.

Video can be sent from the iPhone to the car as well as audio.

Audio can also be sent from the car's microphone to the iPhone.

Once a CarPlay session is established, different combinations of these data streams are used.

When a CarPlay Home screen is first shown, and no audio is playing, the communication protocols are used for setup and location information. An H264 video stream showing the Home screen is sent from the iPhone to the car.

If we then play audio in CarPlay, the communication protocols will additionally be used to send information about the playing track. The video stream here is encoding the now playing screen, and audio for the song is sent to the car.

If the CarPlay UI is exited, and the car's screen is shown, the CarPlay session remains active. The communication protocols will continue to be used for now playing and location information. The video stream is no longer needed, but CarPlay audio still plays, even though CarPlay video is not shown. The screen and audio are controlled independently. Video from the native system can be shown while CarPlay audio is playing. If the CarPlay interface were shown, audio from the native system, like FM radio, could play. Let's see what happens if Siri is then launched using the push-to-talk steering wheel button. The communication protocols will be used to tell the head unit that iOS voice recognition is underway. The CarPlay UI is sent to the screen, and the Siri audio chime plays on the vehicle's speakers. In this case, audio is also sent from the car's microphones to the iPhone.

These examples gave you a high-level overview of how CarPlay works.

Now let's look in more detail at what's happening in the head unit.

For wired CarPlay, data goes over USB.

All data is wrapped in IP. Except for the iAP2 communication protocol.

Information from vehicle sensors is sent over iAP2 as is audio, telephony, and turn-by-turn metadata.

The communication plug-in is source code provided by Apple, and that receives the incoming video and audio streams as well as an additional communication stream.

The audio and video is sent to the head unit's infrastructure, connected to the speakers, microphone, and display.

Control events from the touchscreen, knob, buttons, or touchpad are sent to the plug-in. Now in the case of wireless CarPlay, much of the system looks the same.

Both Wi-Fi and Bluetooth are needed. For wireless, all data is wrapped in IP. Connections to the vehicle's subsystems are the same.

Now that you have a good understanding of how CarPlay works, we'll go over the most important vehicle requirements. We will talk through each of these in detail. We want to make sure you understand the requirements before you begin to design a system so that you can be sure that it's capable of supporting CarPlay.

First, the vehicle needs a great display. A high-quality display is critical to the CarPlay experience.

Several standard display resolutions are supported, and those resolutions are shown here. Most are approximately 16 by 9. Extra-wide screens are also supported. Today, all supported resolutions are landscape orientation.

We have support for screens with varying pixel densities.

Asset sizes in CarPlay have been adjusted depending on the resolution of the screen. So that assets will appear the same size on low and high resolution screens. Without this, icons on the 720 screens would be much smaller.

If you do have a display with a slightly different resolution than what was listed here, please contact Apple to see if it can be supported. We require that aftermarket screens be no smaller than six inches since aftermarket systems could be installed into a wide variety of vehicles. There is no minimum for systems built by automakers. The automaker has the responsibility to make sure that CarPlay is appropriate on the selected vehicle screen.

I should mention several additional screen requirements. Twenty-four bit color depth is required, and a 60 hertz refresh rate is strongly recommended for a responsive user interface. Make sure that your H264 decoder supports the right profile. This is a hard requirement. A high-quality display in the vehicle is critical to the CarPlay experience. Let's talk about the car's audio system and microphones.

CarPlay audio is separated into two streams main audio and alternate audio. Main audio is bi-directional, and it is used for music and other media plus phone calls and Siri. Alternate audio is used for notifications, and it must always be mixed by the head unit with main audio. If the head unit is playing audio, like FM radio, the head unit must mix alternate audio from iPhone with that audio.

LPCM is used for wired CarPlay. Wireless CarPlay requires compressed audio. AAC-LC is used for media, and you have a choice between OPUS and AAC-ELD for other audio.

CarPlay supports the various user input devices in your vehicles.

The CarPlay UI can be controlled by a touchscreen on the head unit, by knobs and buttons, or by a touchpad. The Siri button on the steering wheel is also a critical user input device.

Many vehicles have touchscreens, and let's talk about those first. When the CarPlay UI is active, and the user touches the screen, the head unit sends the iPhone the x, y coordinates of that touch. The head unit sends all touches to the iPhone when CarPlay is displayed on the screen. CarPlay is controlled with single touches only. We support two types of touchscreens, what we call high-fidelity and low-fidelity. High-fidelity touch systems have low latency. So swipe gestures that track the user's finger are allowed. A high-performance system is required, however, one that has less than 140 milliseconds touch latency.

This is how list scrolling using a swipe gesture that tracks the touch input works with a high-fidelity touch system.

Low-fidelity touchscreens may have higher latency and are often resistive screens. Taps control the UI, and a swipe will not track the user's finger.

This is how list scrolling with single taps works with a low-fidelity touch system.

Some vehicles use knob, button, and touchpad controls. Sometimes these are the main user input devices, and sometimes they are secondary to a touchscreen. All combinations are supported by CarPlay.

Knobs are supported by the head unit, sending rotate, select, and back events. This is the minimum set of supported features for a knob that's used as a primary interface. Rotate, select, and back. Some knobs also support a tilt or nudge feature.

This capability is supported by the CarPlay UI.

Vehicle touchpads report x, y coordinates as well as select and a back button.

If a knob or touchpad controller is primary, back and select are required. Secondary touchpads can also support character recognition. Buttons for various functions such as next, previous, and telephony functions are optional.

The CarPlay UI will automatically change to match the user inputs available in the vehicle. All combinations are possible. If the main controller interface is touchscreen, then combinations with a knob, touchpad, or knob plus touchpad are also possible.

A hard button for Siri is required. Siri is really important to the CarPlay experience. The user needs a quick way to always access Siri even if the CarPlay UI isn't showing. Usually this requirement is satisfied with a push-to-talk button on the steering wheel. For the Siri button, we require that the head unit send iPhone all button up/down events so that interaction during the Siri conversation is possible. This wraps up our discussion on user inputs.

Sensor information is important to CarPlay.

CarPlay needs location information from the vehicle so that the map's app works well. Speed and GNSS information can be provided. Every vehicle needs to know its speed. Speed information is used by iPhone for dead reckoning and is critical if the vehicle does not have a GPS or GLONASS receiver. GNSS information includes latitude and longitude if the vehicle has that capability.

Satellite location information from the vehicle is used in addition to the iPhone's sensors to determine the location of the user. This data from the vehicle may be better since a larger antenna can be integrated into the car. This is the recommended configuration since it will result in the best maps user experience. Sending satellite location information to iPhone is required if the vehicle has the technology on board. GNSS information is also required for any system supporting wireless CarPlay since it's more likely that the phone will be in a pocket, bag, or somewhere where the phone itself has poor reception.

What are the requirements for connecting an iPhone to the head unit? Apple CarPlay supports both wired and wireless connections. We're excited about the addition of wireless CarPlay since it offers a very convenient experience for customers. Users can leave their phones in pockets and bags, which is especially nice on a short drive.

Let's first talk about wired connections.

USB is used here, and all audio, video, and control information is sent over USB. Your head unit will need to support a USB role swap. The head unit will switch from a USB host to a USB device. High throughput is a requirement, and you'll want to make sure your USB infrastructure can support the required data rates. If you have multiple USB ports in the vehicle, and not all of them support CarPlay, you'll need to properly label the ports.

Wireless CarPlay is supported in iOS 9.

Both Bluetooth and Wi-Fi are required.

Bluetooth is used for discovery and the initial connection.

Once the Wi-Fi credentials are sent to the iPhone over Bluetooth, Wi-Fi is connected, Bluetooth is disconnected, and all subsequent CarPlay communication is over Wi-Fi. During the CarPlay session, all audio, including phone call audio and video, is transmitted over Wi-Fi as well as the control protocol.

CarPlay requires a Wi-Fi certified AP, and 5 gigahertz is recommended. Note that high throughput, 25 megabits per second, is required. The Bluetooth subsystem needs to support the communication protocol. Wireless CarPlay offers a convenient comfortable user experience, and we're excited to see your vehicles that support it. Now let's hear from Donald about the CarPlay Design Guidelines. Hi. My name is Donald, and I work on design at Apple. Now that you've seen how CarPlay works, let's look at some design guidelines that will help you provide the best CarPlay experience in your cars.

One of the most important things to keep in mind is that CarPlay is really designed to use the entire display.

CarPlay is instantly familiar to iPhone users, and they feel comfortable with it because it's a consistent environment with its own icons, visual language, and system of internal navigation. Every interaction with CarPlay is easier and faster when users can trust that everything on the screen is part of CarPlay.

If a user has to discern which elements are part of CarPlay and which elements are part of your car's user interface, then every interaction becomes more challenging.

Also, your car's native user interface may include icons that resemble or have similar functions to icons within CarPlay. And it can be confusing if these icons are on the display at the same time.

Instead, use the entire display to show CarPlay.

Let's take a look at what should happen when CarPlay starts.

CarPlay always starts the moment iPhone is plugged into your car. Users should never have to think about how to start it.

When the user connects their iPhone to your car for the first time, the car's display should switch to show CarPlay right away.

This provides feedback that CarPlay is started and creates a strong association between the act of connecting and the familiar appearance of CarPlay.

Don't show notifications to announce state changes related to CarPlay. The needed appearance of CarPlay in the display is the most effective visual feedback.

Now let's consider what should happen after the initial CarPlay connection. When users connect their iPhone a second time, it should feel like a continuation of their previous session.

So if CarPlay was showing on the display when the user left the car, CarPlay should be shown again when iPhone is connected. And if CarPlay media was being played when the user left the car, CarPlay media should resume when iPhone is reconnected as long as the user did not select a different audio source between their CarPlay connections.

In other words, we want to always respect the last selection the user made in the car.

Let's see how last user mode should work. Notice that I'm using CarPlay, and it's showing on the car's display. Now if I disconnect, CarPlay ends, and the car falls back to its native user interface. If I plug in again, CarPlay should appear on the display because that was the last thing I was doing in the car.

So if CarPlay was the last thing on the display when it was disconnected, or the car was turned off, CarPlay should be shown when iPhone is connected again. If CarPlay media was playing, CarPlay media should play again.

Let's see a different example. This time, CarPlay is active with the car's display showing the native user interface. Now if I disconnect and then plug in again, CarPlay does not appear on the display because the native user interface was the last thing I was using in the car.

So if CarPlay was not the last thing shown, it should remain in the car's user interface. But even if CarPlay was not the last thing on the screen, if CarPlay media was playing, CarPlay media should play again.

CarPlay is available the moment the user plugs in their iPhone, but they can switch between the CarPlay display and the native user interface at any time. As we saw earlier, the user returns to the car's user interface by selecting your brand's icon in the CarPlay Home screen. They can switch back to CarPlay by using the Apple CarPlay button.

The Apple CarPlay button is the most important indication that CarPlay is available. As soon as CarPlay starts, the Apple CarPlay button must become active in your top level menu.

And the Apple CarPlay must only appear when iPhone is connected and CarPlay is active.

Be sure to hide the Apple CarPlay button when iPhone is disconnected.

And they also show the Apple CarPlay button in an enabled state when CarPlay is active and in a disabled state when CarPlay is inactive.

You can optionally replace an existing button the top level menu with the Apple CarPlay button such as a phone button.

Just remember that the Apple CarPlay button must appear on your car's Home screen whenever iPhone is connected. When iPhone is disconnected, the Apple CarPlay button must disappear or be disabled.

Let's talk about the Apple CarPlay button that should appear in your interface. There are a variety of approved Apple CarPlay logos available to use as buttons in your car's native user interface. Choose the one that makes the most sense with your design.

There's a black logo and a white logo as well as white and black variance with backgrounds. And, finally, there's a color background.

Let's talk about how media sources are handled when iPhone is disconnected.

Let's say that CarPlay is active, and the user is listening to CarPlay audio playback. When iPhone is disconnected, it's a strong indication that they're about to leave the car. Your system should gracefully handle CarPlay media being disconnected.

When iPhone is disconnected, the user should hear silence and audio should not fall back to another audio source. If the iPhone is disconnected while CarPlay is displayed on the car's display, you should gracefully fall back to the last screen displayed before entering CarPlay. Don't show a CarPlay unavailable or device disconnected screen.

So, again, when CarPlay media is playing, and the user disconnects iPhone, they should just hear silence.

The hardware buttons in your car should also be used to support CarPlay.

Siri is an integral part of CarPlay and requires a dedicated tactile button. Because it's so important, the button should be on the car steering wheel. Users expected Siri to work at any time, whether or not CarPlay is currently visible on the car's display. They also expect the Siri button to work like the Home button on iPhone. Pressing and holding it for 600 milliseconds should activate Siri, and the activation delay must not be greater than one second. The media buttons in your car should be connected to CarPlay media playback so users can play, pause, move to the next track or previous track, and fast forward and rewind.

Finally, if your car has shortcut buttons for phone, media, or navigation, they should also be mapped to the respective apps within CarPlay.

Now let's talk about apps that work in CarPlay.

Apps supported by CarPlay will appear on the CarPlay Home screen in the vehicle. Three categories of apps are supported. Audio, those specific to particular vehicles, and messaging apps. If a user has an audio app that's been updated for CarPlay, that app will appear in any CarPlay car the user connects to. The user interface for these apps is fixed with the structure provided by iOS and the content provided by the app itself. You'll notice that audio apps look a lot like the built-in music app. Users choose what to listen to via a list-based UI, and the Now Playing screen shows the appropriate content and controls. The app needs to have a special entitlement in order to be compatible with CarPlay. We've made some improvements in iOS 10 to make the audio apps experience even better. New in iOS 10 is support for tabs, which make it easier to navigate list hierarchies. In the Now Playing screen, you can now see the app name and additional playback control buttons. In the list views, you can tell which item is playing, which are available for streaming, explicit or live stream. Some apps are designed to play audio as soon as they're launched, and we've added support for these apps in iOS 10. All of these combined make for a much improved experience.

Check out the new tabs, the Cloud icon for streaming apps, and the way tend to note what's currently playing.

In the now playing screen, here's the app name and the additional playback controls.

Now let's talk about that next category, apps written to work in specific vehicles.

Apps published by automakers are written to control vehicle functions or give information about the vehicle. They only appear in vehicles specific to that automaker. A CarPlay entitlement is required for these apps as well. New in iOS 10 is Siri support for automaker apps that control vehicle functions. Siri now has built-in support for changing radio stations, changing climate controls, and more. The automaker app communicates with the vehicle to perform these functions after Siri tells the app what the user would like to do. Let me explain how automaker apps work only in the cars that support them. An app icon is enabled in the CarPlay Home screen only if the iPhone is connected to a head unit that supports the same protocol name as the app. A protocol name supported by the head unit and the app need to match. iAP external accessory is used for app to head unit communication.

Head units can declare multiple protocol names in order to support current and future apps with different groups of target cars. Head units should use a protocol name to match all cars from that brand. For example, com.brand. They should also use protocol names to match cars with specific features, for example, electric vehicles or sports models.

A protocol name for the specific model should also be included so that apps that target just that model can be published in the future.

The head unit needs to use iAP2 to declare at least one supported external accessory protocol. Likely more than one. Start and stop external protocol session must also be supported, and external accessory protocol CarPlay must be set to true. If the car supports wireless and wired CarPlay, these iAP2 messages should be implemented on all supported transports. Most automaker apps will communicate with the car to perform tasks such as changing the climate controls. These apps need to design and implement a communication protocol as well.

So how does the app support this feature? The app's Entitlements.plist needs to be modified to include the protocol names that match those on the target head units, as shown here. With these additions plus support for the protocol to communicate with the car, and a CarPlay entitlement, your app will be ready for CarPlay. We're looking forward to seeing some great CarPlay apps from automakers.

We have a new category of apps supported in CarPlay and iOS10, messaging apps. When a messaging app has been updated to support CarPlay, that app will appear on the CarPlay Home screen. Messages can be received and sent, all driven by Siri interactions. We're excited to extend CarPlay capability to messaging apps, making that experience smarter and safer.

We hope this has been a useful overview of CarPlay. CarPlay is a great way to keep the user experience in vehicles fresh long into the future, and we're delighted to get a chance to talk to you about the features that are new in iOS 10.

For more information, check out this session's WWDC URL.

We also have a follow-on CarPlay deep dive video session that may interest you. Thank you for watching.