Testing Tips & Tricks

Hello. Welcome to Testing Tips & Tricks. My name is Brian Croom. My colleague, Stuart, and I are really excited to share some great testing techniques with you that we have been learning recently. As the conference was approaching, we thought it would be really cool to have an app that we could use to find cool things to see and do around the area of the convention center. We've been building this app, giving it views for finding various point of interest around San Jose and listing how far they are away from you. Now, of course, we wanted to make sure we had a really great test suite for this app that we could run to give us confidence that our code was working properly and would keep working as we continue development. Today, we want to share four simple techniques with you that we found really helpful while writing tests for our app. Some strategies for testing networking code in your app, some tips for tests dealing with foundation notification objects, ways to take advantage of protocols when making mock objects in your tests, and a few techniques for keeping your tests running really fast. Let's start talking about networking.

To allow for dynamic content updates, we've been building our app to load its data from a remote web server.

Here are some things that we found useful when writing tests for networking code. But first, quick, a recap from last year. At WWDC 2017's Engineering Testability session, we discussed the pyramid model as a guide to how to structure a test suite, balancing thoroughness, understandability, and execution speed.

In summary, an ideal test suite tends to be composed of a large percentage of focused unit tests, exercising individual classes and methods in your app.

These are characterized by being simple to read, producing clear failure messages when we detect a problem, and by running very quickly, often in the order of hundreds or thousands of tests per minute.

These are complemented by a smaller number of medium-sized integration tests that targeted discreet subsystem or cluster of classes in your app, checking that they worked together properly, each taking no more than a few seconds to run.

And the suite is topped off by a handful of end-to-end system tests, most often taking the form of UI tests that exercise the app in a way very similar to how the end-user will do so on their devices, checking that all the pieces are hooked up properly and interact well with the underlying operating system and external resources. A test suite following this model can provide a comprehensive picture of how an app code's base is functioning.

For testing the networking stack in this app, we really wanted to take the pyramid model to heart and use it as a guide for how to structure our test suite. Here, we see the high-level data flow involved in making a network request in the app and feeding the data into the UI.

In an early prototype of the app, we had a method in our view controller that was doing all of this in a single place, and it looked like this.

The method takes a parameter with the user's location and uses that to construct a URL for a service API endpoint with a location as query parameters.

Then it uses Foundation's URLSession APIs to make a data task for a get request to that URL.

Then the server responds, it would unwrap the data, decode it using foundation's JSONDecoder API, into an array of point of interest values, which is a struct that I declared elsewhere and conforms the decodable protocol. And it stores that into a property to drive a table view implementation, putting it onto the screen.

Now, it's pretty remarkable that I was able to do all of this in just about 15 lines of code, leveraging the power of Swift and Foundation, but, by putting this together in the one method, I the maintainability and especially the testability of this code. Looking at the base of our testing pyramid, what we really want to be able to do here is write focus unit tests for each of these pieces of the flow. Let's first consider the request preparation and response parsing steps.

In order to make this code more testable, we started by pulling it out of the view controller and made two methods on this dedicated PointsOfInterestRequest type, giving us two nicely decoupled methods that each take some values as input and transform them into some output values without any side effects.

This makes it very straightforward for us to write a focused unit test for the code. Here we're testing the makeRequest method just by making a sample and put location, passing it into the method, and making some assertions about its return value.

Similarly, we can test the response parsing by passing in some mock JSON and making assertions about the parsed result.

One other thing to note about this test is that I'm taking advantage of XCTest support for test methods marked as throws, allowing me to use try in my test code without needing an explicit do catch block around it.

Now, let's see the code for interacting with URL session.

Here, again, we pull it out the view controller, made a APIRequest protocol with methods matching the signature of the methods from the request type that we just saw. And this is used by an APIRequestLoader class.

That's initialized with a request type and a urlSession instance. This class has a loadAPIRequest method which uses that apiRequest value to generate a URL request. Feed that into the urlSession, and then use the apiRequest again to parse in your response. Now, we can continue write unit test for this method, but right now I actually want to move up the pyramid and look at a midlevel integration test that covers several pieces of this data flow.

Another thing that I really want to also be able to test at this layer of my suite is that my interaction with the URLSession APIs is correct. It turns out that the foundation URL loading system provides a great hook for doing this. URLSession provides a high level API for apps to use to perform network requests.

objects like URLSession data tests that represent an inflight request. Behind the scenes though, there's another lower-level API URLProtocol which performs the underlying work of opening network connection, writing the request, and reading back a response.

URLProtocol is designed to be subclassed giving an extensibility point for the URL loading system.

Foundation provides built-in protocols subclasses for common protocols like HTTPS. But we can override these in our tests by providing a mock protocol that lets us make assertions about requests that are coming out and provide mock responses.

URLProtocol communicates progress back to the system via the URLProtocolClient protocol. We can use this in this way.

We make a MockURLProtocol class in our test bundle, overriding canInit request to indicate to the system that we're interested in any request that it offers us.

Implement canonicalRequest for request, but the start loading and stop loading method's where most of the action happens.

To give our tests a way to hook into this URLProtocol, we'll provide a closure property requestHandler for the test to set.

When a URLSession task begins, the system will instantiate our URLProtocol subclass, giving it the URLRequest value and a URLProtocol client instance.

Then it'll call our startLoading method, where we'll take our requestHandler to the test subsets and call it with a URLRequest as a parameter.

We'll take what it returns and pass it back to the system, either as a URL response plus data, or as an error. If you want to do test request cancellation, we could do something similar in a stopLoading method implementation.

With the stub protocol in hand, we can write our test. We set up by making an APIRequestLoader instance, configure it with a request type and a URLSession that's been configured to use our URLProtocol.

In the test body, we set a requestHandler on the MockURLProtocol, making assertions about the request that's going out, then providing a stub response. Then we can call loadAPIRequest, waiting for the completion block to be called, making assertions about the parsed response.

Couple of tests at this layer can give us a lot of confidence that our code is working together well and, also, that we're integrating properly with the system. For example, this test that we just saw would have failed if I had forgotten to call a resume on my data task. I'm sure I'm not the only one who's made that mistake.

Finally, it can also be really valuable to include some system level end-to-end tests.

Actually test a UI testing can be a great tool for this. To learn more about UI testing, refer to the UI testing in Xcode session from WWDC 2015. Now, a significant challenge that you encounter when you start to write true end-to-end tests is that when something goes wrong when you get a test failure, it can be really hard to know where to start looking for the source of the problem. One thing that we were doing in our test recently to help mitigate this was to set up a local instance of a mock server, interrupting our UI tests to make requests against that instead of the real server.

This allowed our UI test to be much more reliable, because we had control over the data being fed back into the app.

Now, while using a mock server can be really useful in this context, it is also good to have some tests making requests against the real server. One cool technique for doing this can be to have some tests in the unit testing bundle that call directly into your apps in that work in Stack and use that to direct requests against the real server.

This provides a way of verifying that the server is accepting requests the way that your app is making them and that you're able to parse the server's responses without having to deal with the complications of testing your UI at the same time.

So, to wrap up, we've seen an example of breaking code into smaller, independent pieces to facilitate unit testing. We've seen how URLProtocol can be used as a tool for mocking network requests, and we've discussed how the power of the pyramid can be used to help us structure a well-balanced test suite that'll give us good confidence in our code. Now, I want to call Stuart to the stage to talk about some more techniques.

Thanks.

Thanks, Brian. So, the first area I'd like to talk about is some best practices for testing notifications.

And to clarify, by notification here, I'm talking about foundation-level notifications known as NSNotification and Objective-C. So, sometimes we need to test that a subject observes a notification, while other times we need to test that a subject posts a notification. Notifications are a one-to-many communication mechanism, and that means that when a single notification is posted, it may be sent to multiple recipients throughout your app or even in the framework code running in your app's process. So, because of this, it's important that we always test notifications in an isolated fashion to avoid unintended side effects, since that can lead to flaky, unreliable tests. So, let's look at an example of some code that has this problem.

Here, I have the PointsOfInterest TableViewController from the app Brian and I are building. It shows a list of nearby places of interest in a table view, and whenever the app's location authorization changes, it may need to reload its data. So, it observes a notification called authChanged from our app's CurrentLocationProvider class. When it observes this notification, it reloads its data if necessary, and, then, for the purpose of this example, it sets a flag. This way, our test code can check that the flag to see if the notification was actually received. We can see here that it's using the default notification center to add the observer.

Let's take a look at what a unit test for this code might look like. Here in our test for this class, we post the authChanged method notification to simulate it, and we post it to the default NotificationCenter, the same one that our view controller uses. Now, this test works, but it may have unknown side effects elsewhere in our app's code. It's common for some system notifications like UI applications appDidFinishLaunching notification to be observed by many layers and to have unknown side effects, or it could simply slow down our tests. So, we'd like to isolate this code better to test this. There's a technique we can use to better isolate these tests. To use it, we first have to recognize that NotificationCenter can have multiple instances. As you may note, it has a default instance as a class property, but it supports creating additional instances whenever necessary, and this is going to be key to isolating our tests. So, to apply this technique, we first have to create a new NotificationCenter, pass it to our subject and use it instead of the default instance. This is often referred to as dependency injection. So, let's take a look at using this in our view controller.

Here, I have the original code that uses the default NotificationCenter, and I'll modify it to use a separate instance. I've added a new NotificationCenter property and a parameter in the initializer that sets it. And, instead of adding an observer to the default center, it uses this new property. I'll also add a default parameter value of .default to the initializer, and this avoids breaking any existing code in my app, since existing clients won't need to pass the new parameter, only our unit tests will.

Now let's go back and update our tests.

Here's the original test code, and now I've modified it to use a separate NotificationCenter.

So, this shows how we can test that our subject observes a notification, but how do we test that our subject posts a notification? We'll use the same separate NotificationCenter trick again, but I'll also show how to make use of built-in expectation APIs to add a notification observer.

So, here's another section of code from our app, the CurrentLocationProvider class. I'll talk more about this class later, but notice that it has this method for signaling to other classes in my app that the app's location authorization has changed by posting a notification.

As with our view controller, it's currently hardcoding the default NotificationCenter. And here's a unit test I wrote for this class. It verifies that it posts a notification whenever the NotifyAuthChanged method is called, and we can see in the middle section here that this test uses the addObserver method to create a block-based observer, and then it removes that observer inside of the block.

Now, one improvement that I can make to this test is to use the built-in XCTNSNotificationExpectation API to handle creating this NotificationCenter observer for us. And this is a nice improvement, and it allows us to delete several lines of code.

But it still has the problem we saw before of using the default NotificationCenter implicitly, so let's go fix that. Here's our original code, and I'll apply the same technique we saw earlier of taking a separate NotificationCenter in our initializer, storing it, and using it instead of the default. Going back to our test code now, I'll modify it to pass a new NotificationCenter to our subject, but take a look at the expectation now.

When our tests are expecting to receive a notification to a specific center, we can pass the NotificationCenter parameter to the initializer of the expectation.

I'd also like to point out that the timeout of this expectation is 0, and that's because we actually expect it to already have been fulfilled by the time we wait on it. That's because the notification should have already been posted by the time the NotifyAuthChanged method returns. So, using this pair of techniques for testing notifications we can ensure that our tests remained fully isolated, and we've made the change without needing to modify an existing code in our app, since we specified that default parameter value. So, next, I'd like to talk about a frequent challenge when writing unit tests, interacting with external classes.

So, while developing your app, you've probably run into situations where your class is talking to another class, either elsewhere in your app or provided by the SDK. And you found it difficult to write a test, because it's hard or even impossible to create that external class. This happens a lot, especially with APIs that aren't designed to be created directly, and it's even harder when those APIs have delegate methods that you need to test. I'd like to show how we can use protocols to solve this problem by mocking interaction with external classes but do so without making our tests less reliable.

In our app, we have a CurrentLocationProvider class that uses CoreLocation. It creates a CLLocationManager and configures it in its initializer, setting its desired accuracy property and setting itself as the delegate. Here's the meat of this class. It's a method called checkCurrentLocation. It requests the current location and takes a completion block that returns whether that location is a point of interest. So, notice that we're calling the request location method on CLLocationManager, here. When we call this, it'll attempt to get the current location and eventually call a delegate method on our class. So, let's go look at that delegate method. We use an extension to conform to the CLLocationManagerDelegate protocol here, and we call a stored completion block. Okay, so let's try writing a unit test for this class. Here's one that I tried to write, and, if we read through it, we can see that it starts by creating a CurrentLocationProvider and checks that the desired accuracy and whether the delegate is set. So far, so good. But then things get tricky. We want to check the checkCurrentLocation method, since that's where our main logic lives, but we have a problem. We don't have a way to know when the request location method is called, since that's a method on CLLocationManager and not part our code. Another problem that we're likely to encounter in this test is that CoreLocation requires user authorization, and that shows a permission dialog on the device if it hasn't been granted before.

This causes our tests to rely on device state. It makes them harder to maintain and, ultimately, more likely to fail. So, if you've had this problem in the past, you may have considered subclassing the external class and overriding any methods that you call on it. For example, we could try subclassing CLLocationManager here and overriding the RequestLocation method. And that may work at first, but it's risky. Some classes from the SDK aren't designed to be subclassed and may behave differently. Plus, we still have to call the superclass' initializer, and that's not code that we can override. But the main problem is that, if I ever modify my code to call another method on CLLocationManager, I'll have to remember to override that method on my testing subclass as well. If I rely on subclassing, the compiler won't notify me that I've started calling another method, and it's easy to forget and break my tests. So, I don't recommend this method, and instead to mock external types using protocols. So, let's walk through how to do that. Here's the original code, and the first step is to define a new protocol. I've named my new protocol LocationFetcher, and it includes the exact set of methods and properties that my code uses from CLLocationManager.

The member names and types match exactly, and that allows me to create an empty extension on CLLocationManager that conforms to the protocol, since it already meets all the requirements.

I'll then rename the LocationManager property to LocationFetcher, and I'll change its type to the LocationFetcher protocol.

I'll also add a default parameter value to the initializer, just like I did earlier, to avoid breaking any existing app code.

I need to make one small change to the checkCurrentLocation method to use the renamed property. And, finally, let's look at that delegate method. This part is a little trickier to handle, because the delegate expects the manager parameter to be a real CLLocationManager, and not my new protocol. So, things get a little more complicated when delegates are involved, but we can still apply protocols here. Let's take a look at how. I'll go back to LocationFetcher protocol that I defined earlier, and I'll rename that delegate property to LocationFetcherDelegate. And I'll change its type to a new protocol whose interface is nearly identical to CLLocationManagerDelegate, but I tweaked the method name, and I changed the type of the first parameter to LocationFetcher.

Now I need to implement the LocationFetcherDelegate property in my extension now, since it no longer satisfies that requirement. And I'll implement the getter and the setter to use force casting to convert back and forth to CLLocationManagerDelegate, and I'll explain why I'm using force casting here in just a second. Then in my class' initializer, I need to replace the delegate property with locationFetcherDelegate. And the last step is to change the original extension to conform to the new mock delegate protocol, and that part's easy-- all I need to do is replace the protocol and the method signature.

But I actually still need to conform to the old CLLocationManagerDelegate protocol too, and that's because the real CLLocationManager doesn't know about my mock delegate protocol.

So, the trick here is to add back the extension which conforms to the real delegate protocol but have it call the equivalent locationFetcher delegate method above.

And earlier, I mentioned that I used force casting in the delegate getter and setter, and that's to ensure that my class conforms to both of these protocols and that I haven't forgotten one or the other. So, over in my unit tests, I'll define a struct nested in my test class for mocking, which conforms to the locationFetcher protocol and fill out its requirements.

Notice, in its RequestLocation method, it calls a block to get a fake location that I can customize in my tests, and then it invokes the delegate method, passing it that fake location. Now that I have all the pieces I need, I can write my test. I create a MockLocationFetcher struct and configure its handleRequestLocation block to provide a fake location.

Then I create my CurrentLocationProvider, passing it the MockLocationFetcher. And, finally, I call checkCurrentLocation with a completion block. Inside the completion block, there's an assertion that checks that the location actually is a point of interest. So, it works. I can now mock my classes' usage of CLLocationManager without needing to create a real one. So, here, I've shown how to use protocols to mock interaction with an external class and its delegate. Now, that was a lot of steps. So, let's recap what we did. First, we defined a new protocol, representing the interface of the external class. This protocol needs to include all the methods and properties that we use on the external class, and, often, their declarations can match exactly. Next, we created an extension on the original external class, which declares conformance to the protocol.

Then we replaced all our usage of the external class with our new protocol, and we added an initializer parameter so that we could set this type in our tests. We also talked about how to mock a delegate protocol, which is a common pattern in the SDKs.

There were a few more steps involved here, but here's what we did.

First, we defined a mock delegate protocol with similar method signatures as the protocol that we're mocking. But we replaced the real type with our mock protocol type. Then, in our original mock protocol, we renamed the delegate property, and we implemented that renamed property on our extension.

So, although this approach may require more code than an alternative like subclassing, it'll be more reliable and less likely to break as I expand my code over time, since this way the compiler will enforce that any new methods I call for my code must be included in these new protocols.

So, finally, I'd like to talk about test execution speed. When your tests take a long time to run, you're less likely to run them during development, or you might be tempted to skip the longest running ones. Our test suite helps us catch issues early, when fixing regression is easiest. So, we want to make sure our tests always run as fast as possible. Now, you might have encountered times in the past when you needed to artificially wait or sleep in your tests, because the code your testing is asynchronous or uses a timer.

Delayed actions can be tricky, so we want to be sure to include them in our tests, but they can also slow things down a lot if we're not careful. So, I'd like to talk about some ways that we can avoid artificial delays in our tests, since they should never be necessary. Here's an example. In the points of interest app Brian and I are building, in the main UI, we have a strip at the bottom that shows the featured place. It basically loops through the top places nearby, rotating to show a new location every 10 seconds. Now, there are several ways I might implement this, but, here, I'm using the timer API for foundation. Let's look at a unit test that I might write for this class. It creates a FeaturedPlaceManager and stores its current place before calling the scheduleNextPlace method. Then it runs the run loop for 11 seconds. I added an extra second as a grace period.

And, finally, it checks that the currentPlace changed at the end. Now, this isn't great, and it takes a really long time to run. To mitigate this, we could expose a property in our code to allow us to customize that timeout to something shorter, like 1 second. And here's what that kind of a code change might look like.

Now, with this approach, we can reduce the delay in our tests down to one second. Now, this solution is better than the one we had before. Our tests will definitely run faster, but it still isn't ideal. Our code still has a delay, it's just shorter. And the real problem is that the code we're testing is still timing dependent, which means that, as we make the expected delay shorter and shorter, our tests may become less reliable, since they'll be more dependent on the CPU to schedule things predictably. And that's not always going to be true, especially for asynchronous code.

So, let's take a look at a better approach.

I recommend first identifying the delay mechanism. In my example, it was a timer, but you could also be using the asyncAfter API from DispatchQueue.

We want to mock this mechanism in our tests so that we can invoke the delayed action immediately and bypass the delay.

Here's our original code again, and let's start by looking at what this scheduledTimer method actually does.

The scheduledTimer method actually does two things for us. It creates a timer, and then it adds that timer to the current run loop. Now, this API can be really convenient for creating a timer, but it would help us to make this code more testable if I actually break these two steps apart.

Here, I've transformed the previous code from using scheduledTimer to instead create the timer first and then add it to the current runLoop second, which I have stored in a new property. Now, this code is equivalent to what we had before, but, once we break these two steps apart, we can see that runLoop is just another external class that this class interacts with. So, we can apply the mocking with protocols technique we discussed earlier here.

To do that, we'll create a small protocol, containing this addTimer method. I've called this new protocol TimerScheduler, and it just has that one addTimer method, which matches the signature of the runLoop API. Now, back in my code, I need to replace the runLoop with the protocol that I just created. And in my tests, I don't want to use a real runLoop as my TimerScheduler. Instead, I want to create a mock scheduler that passes the timer to my tests. I'll do this by creating a new struct nested in my unit test class called MockTimerScheduler, conforming to the TimerScheduler protocol.

It stores a block that will be called whenever it's told to add a timer. And with all the pieces in place, I can write my final unit test. First, I create a MockTimerScheduler and configure its handleAddTimer block.

This block receives the timer. Once it's added to the scheduler, it records the timer's delay, and then it invokes the block by firing the timer to bypass the delay.

Then, we create a FeaturedPlaceManager and give it our MockTimerScheduler.

And, finally, we call scheduleNextPlace to start the test, and, voila, our tests no longer have any delay. They execute super fast, and they aren't timer dependent, so it'll be more reliable.

And, as a bonus, I can now verify the amount of timer delay using this assertion at the bottom. And that's not something I was able to do in the previous test.

So, like I said, the delay in our code is fully eliminated using this technique. We think this was a great way to test code that involves delayed actions, but, for the fastest overall execution speed in your tests, it's still preferable to structure the bulk of your tests to be direct and not need to mock delayed actions at all. For example, in our app, the action being delayed was changing to the next featured place. I probably only need one or two tests that show that the timer delay works properly. And, for the rest of the class, I can call the show next place method directly and not need to mock a timer scheduler at all. While we're on the topic of text execution speed, we had a couple of other tips to share.

One area we've seen concerns the use of NSPredicateExpectations. We wanted to mention that these are not nearly as performant as other expectation classes, since they rely on polar rather than more direct callback mechanisms. They're mainly used in UI tests, where the conditions being evaluated are happening in another process. So, in your unit tests, we recommend more direct mechanisms, such as regular XCTestExpectations, NSNotification, or KVOExpectations.

Another testing speed tip is to ensure that your app launches as quickly as possible. Now, most apps have to do some amount of setup work at launch time, and, although that work is necessary for regular app launches, when your app is being launched as a test runner, a lot of that work may be unnecessary. Things like loading view controllers, kicking off network requests, or configuring analytics packages-- these are all examples of things that are commonly unnecessary in unit testing scenarios.

XCTest waits until your app delegates did finish launching method returns before beginning to run tests. So, if you profile and notice that app launch is taking a long time in your tests, then one tip is to detect when your app is launched as a test runner and avoid this work. One way to do this is to specify a custom environment variable or launch argument. Open the scheme editor, go to the test action on the left side, then to the arguments tab, and add either an environment variable or a launch argument. In this screenshot, I've added an environment variable named IS-UNIT-TESTING set to 1.

Then, modify your app delegate's appDidFinishLaunching code to check for this condition, using code similar to this.

Now, if you do this, be sure that the code you skip truly is nonessential for your unit test to function.

So, to wrap up, Brian started by reminding us about the testing pyramid and how to have a balanced testing strategy in your app, showing several practical techniques for testing network operations.

Then, I talked about isolating foundation notifications and using dependency injection.

We offered a solution to one of the most common challenges when writing tests, interacting with external classes, even if they have a delegate.

And we shared some tips for keeping your tests running fast and avoiding artificial delays.

We really hope you'll find these tests useful and look for ways to apply them the next time you're writing tests.

For more information, check out our session webpage at this link, and, in case you missed it, we hope you'll check out Wednesday's What's New in Testing session on video. Thanks so much, and I hope you had a great WWDC. [ Applause ]