キーボードとマウスによるゲームをiPadに

Hello and welcome to WWDC.

Hi, my name is Nat Brown, and I work on the Game Technologies team here at Apple. This video covers adding keyboard and mouse input support to your games for iPadOS.

We've added specific support for keyboard and mouse gaming because we know games are often designed around their main form of input.

For example, most iPhone and iPad games are designed around Multi-Touch input. Multi-Touch input is incredibly flexible. Whole new genres of games have emerged utilizing taps, swipes, touches and gestures. This is Ordia by Loju LTD. It's one of the winners of last year's Apple Design Awards, and it's super fun.

Some games have added on-screen controls with thumbstick areas and buttons similar to a game controller. This has worked well for many first-person view-oriented games, as well as for driving games. Here, I can play one of our sample applications, Fox2, using its on-screen controls. On the left, is a touch area that acts like a left joystick, and on the right, I have two actions, as well as a right joystick.

Still, many games and game genres were really designed around the tactile feedback and the precision of a game controller, and they don't always translate well to on-screen controls. For example, driving games can take advantage of the analog triggers and the shoulder buttons to give finer control to acceleration and braking. And many gamers consider this right thumbstick more sensitive and accurate to use for aiming and camera control. Here in Fox2, I find it more fun to play the game with a game controller on an iPad. It's more of a lean-back experience, and I have additional control. But we also know there are whole genres of very popular games that aren't a perfect fit for touch input and they're also not a perfect fit for game controllers. Multiplayer Online Battle Arena, or MOBA games, using precise mouse control to select areas or to move units, while also supporting keyboard modifiers to those mouse actions, they don't always translate well to using touch input. First-person perspective games that use WASD keys for movement and other left-hand adjacent keys for actions and modifiers, while the mouse precisely aims the camera are another type of game not always perfect for touch. So we've added game-focused APIs for keyboard and mouse input for these types of games. It's a coordinated effort between the UIKit and the Game Controller framework to give you the options you need for creating your best game experience. For the rest of this talk, I'm going to first dive into the details on the new keyboard and mouse APIs in the Game Controller framework. I'm gonna spend a moment describing examples of when you might choose to use the Game Controller APIs versus other pointer APIs at the UIKit layer.

I'm gonna show a quick example of adding support in one of our classic Game Controller samples, Fox2.

And, finally, I'll play a bit of the game to show you the game in action.

So let's dive into the Game Controller framework support for keyboard and mouse input.

The Game Controller framework this year introduces two new classes: GCKeyboard and GCMouse. Like the trusty GCController class, you can find out when these new types of input devices connect and disconnect, register blocks to watch for input changes asynchronously, and, in some cases, you can poll to read the current state of these input devices.

These classes conform to the GCDevice protocol, just like GCController. They have a physical input profile, which you can use to look at the specific input surfaces they expose.

In the case of GCKeyboard, you'll find a set of buttons. And those buttons also appear in the aggregate elements collection. You won't find Dpads or Axes on GCKeyboards.

In the case of GCMouse, you may see zero or more Axes and one or more buttons depending on the type of physical pointing device you have paired at the time. It is exactly the point of physical input profiles to let you tune your game to the specific capabilities of attached devices.

Now we're finally to the actual code, which is my favorite part. I'm gonna show you the types of change handlers you can establish on GCKeyboard devices. And you'll also see how you can inexpensively poll the keyboard key states.

We'll see how to get raw mouse delta events delivered to your change handlers...

and how you use the companion UIKit pointer locking API in order to hide the system pointer and avoid triggering system gestures at the edges of the screen.

We'll also look at how to find out if the keyboard and mouse devices are connected and when they disconnect.

So it's pretty straightforward with GCKeyboard to get notified about key-down and key-up events.

You first check to see if there's a keyboard attached. You'll notice that we're using coalesced on the GCKeyboard. That's because we merge all the keyboard input together. Then, as you are used to elsewhere in the Game Controller framework for watching events asynchronously, you simply assign a change handling block to either the whole keyboard with a key-Changed-Handler, or alternately, assign a change handling block to a specific keyboard button using a value-Changed-Handler. In this case, notice I added a handler to just catch what the space bar is up to or down to.

The other common way to read keyboard input in games is through polling to check the current up or down state of a key rather than through a changed handler. Here we are at some point in our game's update loop, and we're polling the input using this poll-Input function. We are just checking the up/down state of the W, A, S and D keys and reacting by preparing to move our character later in the game update logic.

So whether you choose callbacks or polling is really up to you and your needs, depending on how you are trying to align your input and your rendering loops. One nice thing about these polling APIs is that they are Order(1) and non-blocking, they're non-yielding and they're thread-safe, meaning you can call these APIs without worrying about interrupting or stalling a critical input handling, simulation or rendering thread.

Events coming from the GCMouse devices are not too different from the keyboard. Here you can see how our mouse-Moved-Handler is receiving just delta information coming from the device. This is the delta or change in input value since the last input was received. It is not the current X and Y position of the mouse and screen coordinates, and that's intentional.

With GCMouse, there is no way to poll the current pointer position because, well, you're using this API because you're in charge of defining how and where you want your in-game pointer to appear, if at all.

And you are also in charge of how you want its acceleration to work and the behavior you expect at the screen edges as well.

But what about system gestures, like bringing up the dock, or other multitasking gestures that can happen at screen edges and corners? That takes us to the need to lock the pointer to your application and to hide it. If you're using GCMouse for a full-screen game, you almost certainly wanna do this. Any game that draws its own pointer or which sometimes shows no pointer, a game that performs its own input acceleration, yeah, you'll also wanna lock the pointer from the system's perspective. This will both hide the pointer and it'll lock it to just your full-screen application. The pointer now can't trigger those system gestures that would otherwise break your game.

Telling the system you want the pointer lock is very easy. On your top level ViewController, tell the system if you prefer your pointer locking by overriding the prefers-Pointer-Locked getter to return your current state.

You can dynamically change your mind about having the pointer locked and hidden by simply calling UIViewController's set-Needs-Update- Of-Prefers-Pointer-Locked and returning a new value from prefers- Pointer-Locked when it gets called. This allows you to, for example, use a key to toggle between a mouse-controlled camera mode, where you are controlling your character's viewpoint, and having the system's on-screen pointer, which can be used to interact with user interface elements.

Finally, here's how you detect the presence of a keyboard and mouse. It's just the same as detecting when game controllers connect and disconnect. We keep some local state to keep track of the mouse and maybe whether we've seen a keyboard.

Then, if you wanna have special logic for handling specific mice or multiple mice, you can do so in the Connect-Notification handling block. You might simply set up button and delta event handlers directly here and effectively coalesce any attached GCMouse devices. That's up to you.

In the case of keyboards, you can't differentiate multiple keyboards, and all keyboard events from multiple keyboards are instead coalesced for you. So while you might use notifications to notice when a keyboard disconnects and perhaps pause the game to prompt for different input, in general, you'll find that just using GCKeyboard-Device.coalesced to check on the key state when non-nil is the right path. The coalesced keyboard input is a little bit like the GCController.current controller in this respect-- a lot easier to use if you don't really have any special needs around seeing the device connect or disconnect.

So, when should you use Game Controller APIs for keyboard and mouse handling versus the UIKit APIs introduced in iOS 13.4? The answer is, it really depends, and you might find yourself using a bit of a mix of the two.

If your game is almost always full-screen, has mostly custom UI, renders a custom pointer or hides the pointer during gameplay, wants delta mouse events to implement custom input acceleration and wants to avoid having system gestures intrude on gameplay, or if it needs to poll for the very latest input in multiple time-critical threads, you probably wanna make use of GCKeyboard and GCMouse.

If, on the other hand, your game is only optionally full-screen and can be split screen or multitasking aware...

if the standard system pointer shape morphing and blending support fits your interaction model, and system gestures at the edge of the screen or hot corners do not intrude on your gameplay, and the UIResponder delegate callbacks on your main thread are adequate for receiving keyboard and mouse input, you will have access to all of the new key-up and key-down and key modifier keyboard and mouse events that games are looking for, and you should be fine using UIKit.

The reality is that you may find yourself using both in different modes of your game. During the match portion of a first-person viewpoint game, for example, you may have the pointer locked and no cursor visible, and then you'll be using the delta mouse events. After a match, it may then make sense to unlock the pointer and present standard system UI and a pointer matching the rest of the system. So now let me show you how easy it was to add keyboard and mouse support to our Fox2 sample.

We already had a UIViewController with a lot of functions we need for moving the character, changing the camera and performing actions like jumping and attacking. All we need to do is wire those up to the keyboard and mouse and to keep track of a bit of state.

We're adding a delta variable to pick up the mouse motion which we'll apply in the main game loop.

When we see a new mouse device attach, we're going to call this new registerMouse function to attach our value change handlers.

Now, here where we had that comment to set up our handlers, we first set up the mouse-Moved-Handler to capture those raw delta mouse events we were just talking about. We just store those in our ViewController.

We'll wire up the mouse's left button to cause our fox to fight with the enemies. It's easy because we already wrote those methods for our Game Controller and on-screen touch controls. We're just wiring them to a new input source.

Finally, in our change handlers, we set ourselves up to read from the mouse scroll wheel to change the fieldOfView of the main camera. The camera's already automatically following our character, so we just wire up a fieldOfView change. This is actually a new feature for keyboard and mouse. We didn't have a way to zoom in and out of the touch screen or the Game Controller version.

Last, but not least, we made some small changes in our main update loop. This is the loop of code that runs every frame of the game. We use the most recent delta mouse event data to change our camera direction, and then we clear the delta event variable until the next delta event comes in from the mouse.

For the keyboard, we didn't set up any value change handlers. We're just polling for the current key press state right here in the update loop, and we change the character's direction based on the W, A, S and D keys, like a good little keyboard and mouse game does.

Finally, we allow the user to hold the left Shift key to cause our fox to run instead of walk. That's a pretty common keyboard and mouse game trick-- to use modifier keys to modify motion or aiming.

And that's all it took. Let's compile this new version of the game and give it a play.

So, here I am playing Fox2.

I can use the keyboard. And the W, A, S and D keys allow me to move. The space bar jumps.

I've got control with the mouse of where my camera's looking.

I can use the scroll wheel to zoom in and out.

And that's it for keyboard and mouse gaming on iPadOS. Now you know how to use GCKeyboard and GCMouse and when you might wanna use GCMouse versus UIKit Mouse APIs. You've also seen how easy it was to add to a game that already had controller and touch input.

If you're interested in more ways to control games and game input, you may also find these two other talks useful.

And that's it for keyboard and mouse gaming for iPadOS. Thanks for coming to WWDC.