ReplayKitを使用したMac上でのAppの取り込みとストリーミング

Hello and welcome to W.W.D.C.

Hello and welcome to the ReplayKit for macOS Session. My name is Johnny Trenh and I'm a software engineer on the ReplayKit team. Today I'm really excited to talk to you about ReplayKit on the mac platform as well as some new exciting features for all platforms. Got a lot to cover so let's get started.

Replay kit is a framework that gives you the ability to record or capture your application's screen audio and microphone content to a video that people can edit save and share. ReplayKit also gives you the ability to broadcast your application screen audio and microphone to other third party broadcast services to be viewed all over the world. ReplayKit originally launched for iOS in 2015 and in 2016 ReplayKit for tvOS was introduced. This year I'm pleased to announce ReplayKit is coming to the Mac. All the great features that are available to your iOS and tvOS applications will now be made available for your mac applications. You can now record capture and broadcast your mac application to everyone in the world. The main goal in bringing ReplayKit to the mac was to ensure that you have access to all the great features that we're already shipping with iOS and tvOS.

Along with the same HD quality low performance impact and built in privacy safeguards.

Now as you can imagine applications work quite differently on the mac when compared to iOS or tvOS. So let's take a look at the major ReplayKit features in the mac and I'll start with screen recording. Screen recording gives you the ability to record your application screen audio and microphone into a video that people can then edit save or share. How does screen recording work under the hood. Well let's take a look. Your application will call into RPSScreenRecorder to get the sharedRecorder singleton instance. With the shared instance you can start Recording at which point the Replay Kit will start to capture the screen audio and microphone samples from your application and start writing those samples into a movie file. When your application calls stopRecording replay kit saves the recording calls a preview and share extension and passes that information backup through RPPreviewViewController. You then present the preview view controller which allows people to edit save or share that movie file. Why don't we jump in and see how we can start in app recording in a project.

So here I want to talk a little bit about our project first. I'm here on our main story board. So the main story board we have three buttons that are going to indicate the different functionalities that we're going to express and build out throughout this demo. We're only going to focus on it start recording button right now that you see right here on our storyboard. So this button is already hooked up to our code via IBOutlets and through the storyboard. What we're going to look at now is the IBAction for what happens when someone presses this button. So here in our code the IBAction is right here for recordButtonTapped button tapped. Now we're gonna do two different types of things with this button. Were actually going have the button pull double duty for us if we're actively recording. This button is going to stop recording if we're not actively recording. This button is going to start recording for us. So let's go ahead and take a look at what happens when we start recording. So here is our IBAction and we're going to look at the method start recording. Now it's very easy. The first thing you want to do is you want to go ahead and call RPScreenRecorder and get the sharedRecorder instance. From there we're gonna go ahead and call startRecording. After that this entire block is going to be executed right here. So the first thing you want to do because we're gonna give back an error is you want to see if we have an error. Now if we don't have an error then we're properly recording and everything is started up just fine. So here we say that there's no error. All we're simply going to do is we're gonna go ahead and update our recordingState. Now what do we do when we update our recordingState. Actually what we do when we update our recordingState is we simply set the button to reflect that. Are we recording or not recording. Start or stop. And that's it. And we update our state. After that we are now up and recording. ReplayKit is now actually capturing everything and recording a video for you. So what happens when your user taps the button again. Well here we're going to go ahead and call stopRecording. And let's take a look at how that works. Going down to the stopRecording method. Right here once again we're going to go ahead and get the sharedRecorded instance and then we're going to go ahead and call stopRecording. When we do that to completionHandler block is then executed. Now this looks a little bit complicated but I'll walk you through it because it's actually really simple. The first thing when you do is check to see if we have an error if we don't have an error. Let's check to see if we have a previewViewController.

Now this is where it gets really interesting. But we want to do with the previewViewController is like we spoke about you want to display it so people can edit save or share their video. So we get the previewViewController.

We go ahead and save a reference to that here and then this is the important part. You want to make sure that you are a delegate of the previewViewController. And I'll talk to you about why that's important just a little bit. Then you want to go ahead and share the sheet. In this case I'm using main window and begin sheet. You can share the sheet or show the sheet however which way you choose using UIKit. But in this example we're gonna go ahead and use begin sheet and that's it. We're gonna go ahead and show the sheet.

We do some error checking here. And if there's errors we print them. If not then after that we go ahead and set the recordingState because we're no longer recording. So let's take a look at what happens after the user interacts with the previewViewController. That's why we want to be part of the delegate. So I'm going to go ahead and scroll down here. And here is our delegate previewViewController did finish. This is going to be called when the user is done using the previewViewController. They're gonna go ahead and click save or exit out of it. And that's when you get notified. So the reason why this is important is because we want to keep a reference to that PreviewViewController so that we can dismiss it.

And that's it. Just like that you're up and running with ReplayKit and recording your application to a video. Now that you've seen what it looks like in code I want to take a moment to highlight some of the differences with RPPreviewViewController. People will be able to edit and trim the recording by tapping on the edit button in the preview view controller. Saving the video will be a very familiar experience for people as it follows the standard macOS file save flow. With just two simple API calls. You're now up and running and recording with ReplayKit with in app screen recording you and your application don't have access to the movie file. ReplayKit handles the creation of the movie and people drive the sharing and saving of the recorded session. However in the past I've received a lot of requests from other developers about having access to the recorded movie. I think we finally have an answer for those requests. This is how screen recording currently works. As you can see the movie file is created and saved within ReplayKit and your application has no access to recorded movie itself.

It is only provided with a preview view controller for people to edit and save the video. But what if instead of ReplayKit handling the video file.

We give your application to read access to the video. You can now do that with our new stop API. The new stopRecordingWithURL will allow you to provide a destination URL and ReplayKit will write and save a recorded movie to that URL. You now have direct access to the recorded movies and you can incorporate them into your application. With it you can create and manage replay videos in your application. You can even create a custom video editor and have it integrated right into your own app experience.

These are just some of the new and exciting experiences you can create using the newStopRecording AP but you might be thinking what if I don't want ReplayKit to make the movie recording for me. What if I want more control of my application's video and audio content. And for that you can use In-App screen capture the next major feature we've brought to the Mac for ReplayKit. With in app screen capture Replay Kit will send the audio and video samples straight to your application's process where you'll have complete control in how to use them. Let's go ahead and take a quick look at how that works. You've already seen how in app screen recording is done by interacting with the shared screen recorder instance. But instead of calling startRecording. When startCapture is called on the sharedRecorder ReplayKit will start to capture your audio visual and microphone samples from your application. However instead of ReplayKit creating and managing a movie for you as we do in recording we sent all audio and visual samples back to your application process. Let's jump back into our sample code project and see how we get started with that capture for the Mac. OK. So we're back into our project here and once again we're taking a look at the storyboard. Remember like we said before we're building out this demo so that we can cover all three of the major functionalities in ReplayKit. We've already covered the start recording button. Now we're going to go ahead and focus on the start capture button. Let's take a look at how that looks like inside the code. So here. We have IBAction that's tied to the button from our storyboard. Just like with the recording our button is gonna pull double duty if we're actively not recording or broadcasting or capturing. Tapping this button is gonna start capture.

Now if we are actively capturing tapping this button is gonna stop capture. So let's go ahead and take a look at what happens when we hit startCapture.

Here's our code for startCapture. Now it looks a little bit crazy but I'll talk to you step by step here we're gonna go ahead and call screenRecorder and get the sharedRecorder instance. From there you're gonna call startCapture.

startCapture is gonna take in two different handlers. The first handler is going to be your sample handler which we see here. This block of code is executed every time ReplayKit gives you a sample whether it be audio video or microphone. So this block of code is run continuously. Now the second handler is going to be the completiionHandler which we have here.

Now the completionHandler is called only once when you start capture. This is much like the recording completion handler where it signifies to you that capture has started and we'll give you an error if there's something that happened in the process. So let's take a look at the completion handler first and then we'll take a look at what we do when the samples come in.

Here in the completion handler just like with recording we're going to check to see if we have any errors if we don't have any errors. We're going to set the caputre state much like we did with recording. Well what do we do in the recording capture state. Well we set the button title our active state internally and we disable the other buttons and we set up the camera view. Now we saw this back in the recording as well. The camera view is going to allow you to use the camera pip to add to your application.

So what happens if we do have an error. We're going to go ahead and print the error and move forward. Now let's go ahead and look at the sample block because this is actually the most interesting bit because this gets called every time ReplayKit gives you a sample. So what you want to do is you want to go ahead and take a look at all the different samples and all the different types that we give back. Then you want to go ahead and decide what you want to do with it. So in this completion block we give you back a sample. We give you back the sample type and an error. And in this block right here the first thing we do is we take a look to see if you have any errors. If we do we go out and print them. If not we're going to run a switch statement on the type. Now I'm sure in your application you're going to be doing a lot of really really awesome things but in this application we'll keep it simple. We're gonna go ahead and go through each one of these types and we'll process them separately. So for if it's a video type we'll go ahead and say processAPPVideoSample. We'll do the same for audio and mic And that's it. These samples are given to you from ReplayKit and then you can do whatever you need to do with them. Let's take a look at what we do in this example. Which is pretty simple for all of our process methods all we do is print. Now again I'm sure you can do something far more interesting and I'm excited to see what you do with it. But for this example we just print and that's it. We're getting all of these samples and we print if we get the audio we print if we get a video and then we print if we get a mic.

And just like that you're up and running with capture. With in app screen capture your application now has access to all the video and audio samples through ReplayKit. You can build dynamic in app experiences that include screenshots now that you have access to the samples you can add custom gaming overlays to the videos. With this raw data you can even implement a custom heads up display like this for your application. The possibilities really are endless.

OK so now we can record and we can capture but what happens when you're ready to take things global. That's what live broadcast is for. The final major feature we've bought over to macOS for ReplayKit. With live broadcasts you give people the ability to stream their experience and your application to a third party streaming service allowing viewers from all over the world to watch live. Let's take a quick look at how live broadcast works before we jump into another example of how to set up a live broadcast. With in app broadcast you will call into RPBroadcastActivityController to display a picker for the available third party streaming services. When people select a third party streaming service ReplayKit will setup the broadcast connection to the streaming service and give back a RPBroadcastController to your application. The broadcast controller is what you'll use to control the actual broadcast. Here's a quick overview of the three part process that'll allow you to start live broadcasting your application. People will initiate a broadcast and you'll present the broadcast picker. People will select the broadcast service and you'll get back a broadcast controller.

Then you'll start to broadcast using that controller.

Now that we have an idea of how that works let's go ahead and take a look at how we can do that in a project. OK so we're back live here in our storyboard. We've already covered the start recording and start capture. Now we're going to go ahead and focus on the start broadcast button. This is where things get really really cool. All right let's jump back into our swift code here and here we go. We're right here on the IBAction for the button tap. Once again our broadcast button is going to pull double duty for us if we're active. The broadcast button is going to start broadcasts. And if we're not active it's going to start our broadcast. Let's look at the more interesting bit which is starting a broadcast and you'll see how easy it is to actually get your app global and broadcasting to the world. All right. Here is our button. And we're going to go ahead and do the presentBroadcastPicker like we covered in the slides. This is what happens when your application presents a picker for the available third party broadcasts so that people can choose. Let's go ahead and jump into that code right here and here it is. Well first thing we're going to do is we're going to decide on a picker origin point. This is the origin point. That is the bottom left hand corner of the broadcast picker. Next we're going to go ahead and go on to the RPBroadcastActivityController and we're going to call showBroadcastPicker at our origin point from the shared window of our application. And we're going to pass in a nil for preferred extension because for now I just want to show all the available third party streaming services. All right. When we call that this block right here is going to get executed. And the first thing we would do is check for errors if we don't have an error. This is what we're going to do. We're going to go ahead and set the activity controller that we give back from this call and we're going to keep a reference to it. You'll see why this is important later. And then you're going to get a activity controller delegate. Now the delegate part is a really really important part. You're going to need to set yourself as the delegate or your application as the delegate. I will explain that in just a little bit.

If we have an error go ahead and print it. So now you're gonna go ahead and present this picker. People are going to go ahead and choose a broadcast that they want to do. And then you're going to want to be notified when they're done. This is where the delegate comes in and why it's very very important. Let's go ahead and take a look at the delegate call. Here is the BroadcastActivityController delegate. Did finish with broadcast controller. So we definitely want to go ahead and be a delegate here. And the reason why is when the user is done picking a third party broadcast ReplayKit will connect your application to the broadcast extension. And we're gonna give you back a broadcast controller. You need the broadcast controller here because that broadcast controller is what you're gonna use to actually control the broadcast. So in this call we're gonna go ahead and check to see if we have an error. We don't have an error. We're gonna go ahead and save the broadcast controller because the broadcast controller again controls your broadcast. You can pause. You can stop start. All of those things are tied in with the broadcast controller. Then we immediately use a broadcast controller to call startBroadcast. Then another block is executed. This is just like your startRecording. You call this and we give you back an error. If there's an error that means something went wrong but if there isn't you are now broadcasting to the third party streaming service people have selected.

We do is update our broadcast state like we did it the other ones which is going to update our button title our internal state and disable the other two ReplayKit functional buttons. And that's it. You're now broadcasting. You are now sending audio and video to the users selected third party broadcasts service. Now what happens when we want to call stopBroadcast. Now this is interesting because when we call stopBroadcast right here. This is why we want to save the broadcast controller because here we simply call our reference broadcast controller and we call finishBroadcast at which point this entire block will be executed just like with stopRecording and stopCapture. We'll give you an error if something has happened. First thing you want to do is check the error. We don't have an error set your broadcast date to false because we're no longer broadcasting. Turn down your camera and anything else that you need to do. And that's it. You've just started and stop a live broadcast with their application and you're ready to take your application global.

I want to take a minute to go into a little bit more depth what the differences for in app live broadcast that were made for macOS. If you're coming from iOS. you'll notice that we no longer interact with the RPBroadcastActivityViewController class. Instead for macOS that class has been replaced with RPBroadcastActivityController. Due to macOS supporting multiple screens and windows instead of calling loadBroadcastActivityViewController like we do in iOS for macOS we call showBroadcastPicker with a CGPoint and a window reference. We also take in a preferred extension identifier if your application wants to stream directly to a specific broadcast service. The CGPoint passed in is a reference to the origin point of the passed in window and represents the bottom left hand corner for the broadcast picker. You're now global with your application using ReplayKit live broadcast. When you start to implement ReplayKit in your macOS applications you're going to notice something new that's been added to your menu bar. With ReplayKit on macOS in keeping with the goal of user privacy you'll see a new menu bar item shown on the menu bar every time you start an active recording capture or broadcast session. The menu bar icon indicates that there is an active ReplayKit session. It also serves as a way for people to stop the active ReplayKit session. You'll need to make sure that you adhere to the rpScreenRecordingDelegate method call didStopRecordingWithPreviewViewController as this will be called when people click on the menu bar icon to stop a session. For in app capture and live broadcast sessions the RPPreviewViewController passed in will be nil. With that you're now ready to add ReplayKit to record capture and broadcast your applications and share them with the rest of the world. But we have one last thing we want to talk about and that is game controllers. I'm happy to announce that there is now ReplayKit support for the Game Controller Framework and iOS tvOS and macOS. The game controller framework will now have built in ReplayKit functionality. Double tapping the share button on the PS4 controller or the select button on Xbox controller will start an in app recording. Double tapping again will automatically stop the in-app recording and save it to your photos for iOS and your desktop for macOS.

For applications already using ReplayKit game controllers can start and stop recording's outside your Applications control. So it's a good idea to make sure you're using key value observing for both availability and recording properties on RPScreenRecorder so that you can update your state as needed.

Also be sure to follow the protocol method didStopRecordingWithPreviewViewController on RPScreenRecorderDelegate so that you can update your application state as needed. I'm super excited to be announcing ReplayKit support for game controllers. I hope to see it in your games very soon.

Wow we've covered a lot today let's go ahead and take a quick recap of what we talked about. I covered in app screen recording where ReplayKit will record your applications audio and video into a movie that people can edit share and save. I covered in app screen capture where ReplayKit gives your applications audio and video samples right back to you in your applications process. I covered in app screen broadcast where ReplayKit sends your application audio and video to a third party live streaming service. And finally I covered the new built in functionality found in GameControllers Framework.

The sample code project as well as the documentation regarding ReplayKit Framework or to revisit this session please visit us at developer.apple.com We look forward to seeing you add ReplayKit to your iOS tvOS and now macOS applications.

Thank you so much for watching our session on ReplayKit for macOS and I hope you have a wonderful WWDC