What's New in Testing

Good morning. Welcome to Session 403, What's New in Testing. My name is Honza Dvorsky and I will share the stage today with my colleague, Ethan Vaughan.

Today, we'll start by talking about the code's coverage improvements we made in Xcode 9.3.

Then we'll look at the test selection and ordering features new in Xcode 10. And finally, Ethan will come up on stage to tell you about how you can speed up your tests by running them in parallel. Let's kick off with code coverage. We completely reworked code coverage in Xcode 9.3 and this resulted in many performance and accuracy improvements and it allowed us to add a new feature so that you have the granular control over which targets contribute to code coverage. We created a new command line tool called xccov and last but not least, we gave code coverage in the source editor a visual refresh. Let's look at all of these in detail. First, to give you a sense of how much better code coverage is now, we measured it on a large internal project at Apple. To understand the speed, we measured how long it took Xcode to load and show code coverage in the source editor. In Xcode 9, it took about six and a half seconds to do that. In Xcode 9.3, however, we got it down to less than a half a second, which is more than 95% faster.

But we also wanted to make the coverage file smaller, as Xcode can potentially write out many of them, if you run your test often, as you should, so I'm happy to say that here, the improvements are just as dramatic. Xcode 9's coverage files were over 200 megabytes in size. And this is a lot but remember that we're talking about a large project with thousands of source files. But files written by Xcode 9.3 were less than a tenth of that. I'm sure you appreciate this if you maintain a continuous integration machine or if you're just running low on disk space. But the best part is that not only are the coverage files smaller and faster to read and write, they're also more accurate than ever before. One example of this is header files. Xcode 9 didn't correctly collect and show coverage in header files and this was a problem for a code basis that used C++ as in that language, you can have a good portion of your executable code in headers.

So, if you're one of the people affected, you'll be happy to hear that Xcode now correctly gathers and presents code coverage for both implementation and header files.

Now let's talk about code coverage features.

The first one is target selection.

This is a new option to control not only whether code coverage is enabled or disabled but when it's enabled, you actually control which targets are included.

This can be important if your project builds third-party dependencies that you're not expected to cover by your tests or if you work at a company where another team supplies you with a framework that they already tested. You can customize the included targets in this scheme's test action under options.

This means that it can continue to include all targets in code coverage or only hand pick some of them.

Now in the name of more powerful workflows, we created a new command-line tool called xccov. It can be integrated into automation scripts easily, as it produces both human readable and machine parseable outputs.

And at the high level, it gives you detailed view of your coverage data. So, I've mentioned the code coverage data a couple of times, let's look at how it looks under the hood. When tests are run with code coverage enabled, Xcode generates two files. First is the coverage report, or the xccovreport file extension, and that contains the line -- line coverage percentages for each target, source file, and function. The second one is the coverage archive and that contains the raw execution counts for each of the files in the report.

And these coverage files live in your project's derived data directory and additionally, if you pass the result bundle path flag to Xcode build, these files will be placed into the result bundle as well. Let's look at an example. Here, we use xccov to view the coverage data of our Hello World app. We can see the overall coverage for each target but also detailed coverage for each file and even for each method.

And of course, by passing the right flag identification, you can get the same exact information in JSON. This can make it much easier to integrate with other tools.

So, as you can see, xccov is very flexible, so I would encourage you to explore its documentation. Now we talked about use -- viewing code coverage on the command line but the most convenient way is still going to be to see it right next to your source code. You control whether code coverage is shown by selecting editor, show, or hide code coverage.

When you do enable it, you'll see the refreshed look of code coverage in the source editor. The whole line highlights when you hover over the execution count on the right-hand side.

And this is just the best way to keep an eye on which of your code is already covered by tests and which still needs more work. Now let me show you all of these improvements in action.

So, here I have a project called Dev Cast. It's a simple messaging app and I created the iOS version for this talk last year. So, this year, I wanted to create a Mac version. But I wanted to share the business logic between the iOS and Mac versions, so I put it all into a framework called DevCastkit and today, my goal is to explore and maybe improve the code coverage of this framework.

So, right now, I'm not collecting code coverage, so I just need to turn it on. I'll do that by selecting the scheme, selecting edit scheme, go into the test action, go into options. And down here, I will just enable code coverage.

Now I will run my test by going to product test. Right now, my framework, my test bundle and my Mac app are getting built, and my tests are being run. Now the test finished so let's look at the results. We'll go to the report navigator and to the latest coverage reports.

Here, I can see that I have the -- both targets, the Mac app and also the framework. Now the Mac app only serves as a container for running tests, so I'm not really concerned about its low code coverage. In fact, I would actually not want to see it in this report at all. So, I can do that by editing the scheme again. And in the test action, here -- instead of gathering coverage for all targets, I'll change to just some targets.

This gives me a hard cover list and I will add just the target that I'm interested in, which is the framework.

Now I will rerun my tests and look at the updated coverage reports.

So, now you can see that I only have the framework in the coverage report. This is what I wanted. So, now I can just focus on this one target. So, looking at the coverage percentage, is at 84%, which is not bad, but I know I can do better. So, to understand which of the files need more attention, I will disclose the target and see that the first file is only covered at about 66%. So, I'll jump to the file by clicking on the arrow here. And here, I'll look at my local server class. On the right-hand side, I can see the execution counts for each region of the code so I can see that all of the executable parts of my class are covered by tests. This is good. Unfortunately, my convenience function, get recent messages, hasn't been called for my tests at all, so I don't know if it's working properly.

To fix that, I'll go to the corresponding test file and I will add a new test.

This test just puts a couple of messages into a local server and then I call, get recent messages, on it to verify that it returns what I'm expecting.

So, with the new test added, I will rerun my tests one more time.

So, great, the test finished. I'll go to the coverage report again and now when I focus on my framework, I can see that it's covered at 100%. This is exactly what I wanted.

So, we saw how I used the target selection feature to only focus on some of the targets. Then I used the coverage report to know exactly which file to focus on. And finally, I used the code coverage integration in the source editor to know exactly which part of my code still needs covering. So, this is the improved code coverage in Xcode.

So, moving on from code coverage, let's talk about some new features in Xcode 10. First, we'll see how we can now better select and order our tests. Now why is this important? Well, not all the tests in your suite serve the same purpose. You might want to run all 1000 of your quick running unit tests before every single commit but only run your 10 long-running UI tests at night. And you can achieve this today by disabling specific tests in your scheme.

The scheme encodes the list of disabled tests so that XE test knows which tests to skip. And this has an interesting side effect. Whenever you write a new test, it's automatically added to all the schemes that contain the corresponding test targets. But if that's not what you wanted, you have to go through all those schemes and disable the test there manually.

So, in Xcode 10, we're introducing a new mode for schemes to instead, encode the tests to run.

If you switch your scheme to that mode, only the test that you hand pick will run in that scheme. You control this mode in the scheme editors test action, where in the list of test targets, there's a new options pop-up, where you can declare whether the scheme should automatically include new tests or not.

This way, some of your schemes can continue to run any new tests you write, while other schemes will only run a list of hand-picked tests. So, we've discussed how we can better control which of our tests run and when, but the order of our tests can be significant too.

By default, tests in Xcode are sorted by their name. This means that unless you rename your tests, they will always run in the same order. But determinism can be a double-edged sword. It can make it easy to miss bugs, where one of your tests implicitly depends on another one running before it. Let's look at an example of when this can happen.

Imagine you have tests A, B, and C. They always run in this order and they always pass. But when you look at your tests in detail, you realize that test A creates a database.

Then test B goes and writes some data into it. And then finally, test C goes and deletes it.

Now these tests only pass because they run in this specific order. But if you tried to shuffle them around, for example, by renaming them, then you try to run them again, you might have test B writing into a database that doesn't exist and your tests would just fail.

So, to prevent issues like this, your tests should always correctly set up and tear down their own state.

Not only will they be more reliable, but it will also make it possible for your tests to run independently of all the other ones and this can be really beneficial during development and debugging. So, to help you ensure that there are no unintentional dependencies between your tests, in Xcode 10, we're introducing the new test randomization mode. If you turn it on, your tests will be randomly shuffled before every time they're run. And if your tests still pass with this mode on, you can be more confident that they are really robust and self-contained.

Randomization mode can be enabled in the scheme editor, just like the other features we've seen today. So, these are the new test selection and ordering features in Xcode 10. Now I'm really excited about what comes next.

To tell you all about the new parallel testing features in Xcode, I would like to welcome Ethan Vaughan to the stage. Ethan. Thanks, Honza.

So, many of you have a development cycle that looks like this. You write some code, you debug it, and then you run your tests before you commit and push your changes to the repository.

By running your tests before you push, you can catch regressions before they ever make it into a build.

However, one of the bottlenecks in this process can be just how long it takes to run your tests. Some of you have test suites that take on the order of 30 minutes to hours to run. And if you have to wait that long before you can confidently land your work, that represents a serious bottleneck in your workflow.

We want to make sure that your tests run as fast as possible in order to shorten this critical part of the development cycle. So, last year, we introduced a feature in Xcode 9 to help you run tests faster, it's called Parallel Destination Testing. This is the ability to run all of your tests on multiple destinations at the same time. And you do this from the command line by passing multiple destinations specifiers to xcodebuild.

Previously, if you were to run your tests, let's say on an iPhone X and an iPad, xcodebuild would run all of the tests on the iPhone X and then all of the tests on the iPad. Neither device would be running tests at the same time. But in Xcode 9, we changed this behavior so that by default, tests run concurrently on the devices and this can dramatically shorten the overall execution time, which is great. However, there are some limitations to this approach.

First, it's only beneficial if you test on multiple destinations. If you just want to run unit tests for your Mac app, for example, this doesn't help. Also, it's only available from xcodebuild, so it's primarily useful in the context of a continuous integration environment, like Xcode Server or Jenkins.

I'm excited to tell you about a new way to run tests faster than ever called Parallel Distributed Testing.

With Parallel Distributed Testing, you can execute tests in parallel on a single destination. Previously, testing on a single destination looks like this, a continuous straight line with one test executing after the other.

Parallel Distributed Testing allows you to run tests simultaneously so that testing now looks like this.

In addition, it's supported both from Xcode, as well as xcodebuild, so no matter where you run your tests, you'll get the best performance. Now in order to tell you about how Xcode runs your tests in parallel, we first have to talk about how your tests execute at all, what happens at runtime. Let's start with unit tests. Your unit tests get compiled into a test bundle.

At runtime, Xcode launches an instance of your app which serves as a test runner.

The runner loads the test bundle and executes all of its tests, so that's how unit tests execute. What about UI tests? For UI, tests the story is similar. Your tests still get compiled into a bundle, but the bundle is loaded by a custom app that Xcode creates. Your app no longer runs the tests.

Instead, the tests automate your app by launching it and interacting with different parts of its UI.

If you'd like to learn more about this process, I'd encourage you to check out our session from 2016, where we go into even more detail. So, now that we understand how our tests execute, we can finally talk about how Xcode runs them in parallel. Just like before, Xcode will launch a test runner to execute our tests but instead of just launching a single runner, Xcode will launch multiple runners, each of which executes a subset of the tests. In fact, Xcode will dynamically distribute tests to the runners in order to get the best utilization of the course on your machine. Let's go into more detail.

When Xcode distributes tests to the runners, it does so by class.

Each runner receives a test class to execute and it'll execute that test class before going on to execute another one. And then testing finishes once all the classes have executed. Now you might be wondering why Xcode distributes tests by class instead of distributing individual test methods to the runners.

There are a couple reasons for this. First, there may be hidden dependencies between the tests in a class, like Honza talked about earlier. If Xcode were to take the tests in a class and distribute them to different runners, it could lead to hard to diagnose test failures. Second, each test class has a class level set up and tear down method, which may perform expensive computation. By limiting the tests in a class to a single runner, XE tests only has to invoke these methods once, which can save precious time. Now I'd like to talk about some specifics to parallel testing on the simulator.

When you run tests in parallel on the simulator, Xcode starts by taking the simulator that you've selected and creating multiple distinct copies or clones of it.

These clones are identical to the original simulator at the time that they're created.

And Xcode will automatically create and delete these clones, as necessary.

After cloning the simulator several times, Xcode then launches a test runner on each clone and that runner then begins executing a test class, like we talked about earlier. Now the fact that your tests execute on different clones of the simulator has some implications that you should be aware of. First, the original simulator is not used during testing. Instead, it serves as a sort of template simulator. You can configure it with the settings and the content that you want and then that content gets copied over to the clones when they're created. Next, there will be multiple copies of your app, one per clone, and each copy has its own data container.

This means that if you have a test class that modifies files on disk, you can't expect those file modifications to be visible to another test class because it could have access to a completely separate data container. In practice, the fact that class is executed on different clones will likely be invisible to your tests, but it's something to be aware of. So, where can you run tests in parallel? You can run unit tests in parallel on macOS, as well as unit and UI tests in parallel on the iOS and tvOS simulators. And with that, I'd like to give you a demo of Parallel Distributed Testing in action.

All right, so this is the Solar System app, which you may have seen at some of the other sessions here at WWDC. As one of the developers of this app, I want to run all of my tests before committing and pushing my changes to the repository.

However, as I add more and more tests, this is starting to take longer and longer, and it's becoming a bottleneck in my workflow. So, let's see how parallel testing can help. I'll go ahead and switch over to the Xcode project and I already ran my tests and I want to see how long they took to run. All right, let's go back to the test report. So, here, you can see next to each method, we show exactly how long that method took to execute.

Next, for each test class, we show the percentage of passing and failing tests, as well as how long the test class took to execute.

And finally, in the top right corner, you can see exactly how long all of your tests took to run. So, here, we can see that our tests took 14 seconds.

Now I'd like to enable parallelization, which I can do by going to the scheme. So, I'll select the scheme and choose edit scheme. Next, I'll click the test action and I'll click the options button next to my test target.

Finally, I'll select the execute in parallel checkbox and that's all I need to do to enable parallel testing. So, let's go ahead and run our tests now by choosing product tests. So, I want you to look at the doc. Xcode launches multiple copies of our Mac app to run our unit tests in parallel. So, great, it looks like that finished. So, now, let's go back to the report and select the most recent report.

So, whereas previously, our test took 14 seconds to run, now they only take 5 seconds. So, just by enabling parallelization, we've seen the greater than 50% improvement in test execution time.

So, the Solar System app isn't just available for the Mac, it also has an iOS companion. And one of my responsibilities was to write a UI test suite to exercise the various screens of my iOS app. Now I already enabled parallelization for the iOS scheme, so I'll go ahead and just switch to that now. And then I'll run tests by choosing product, test. Now we'll go ahead and switch over to the simulator. So, here, you can see that Xcode has created multiple clones of the simulator that I selected, and these clones are named after the original simulator so that they're easy to identify. And on each simulator, Xcode has launched a test runner and each runner is executing a different test class in my suite.

Now while my tests are executing, I'm going to switch back to Xcode and show you the test log. You can find the test log associated with the test report.

The log is a great place to see how your classes are being distributed to the different runners. You can see an entry in the log for each runner and beneath a given runner, you can see the exact test class that it is currently executing.

So, when the tests are completely finished, this is a great place to see how your classes were distributed, which gives you a picture of the overall parallelization. And with that, let's switch back to the slides.

So, let's briefly recap what we learned in the demo.

First, we saw how to enable parallelization in the scheme editor.

Next, we saw how to view results in the test report, as well as how our classes are distributed in the test log.

Then we saw Xcode launch multiple instances of our Mac app to run unit tests in parallel.

And finally, we saw multiple clones of the simulator running our UI tests in parallel. Like I mentioned earlier, xcodebuild has great support for parallel testing as well. We've added some new command line options that allow you to control this behavior and I'd like to point out two of those now.

First, we have parallel-testing-worker-count, which allows you to control the exact number of workers or runners that Xcode should launch during parallel testing.

Normally, Xcode tries to determine an optimal number of runners based off of the resources of your machine, as well as the workload. This means that on a higher core machine, you will likely experience more runners.

But if you find that the default number isn't working well for you, you can override it using this command line option. Next, we have parallel-testing-enabled, which allows you to override the setting in the scheme to explicitly turn parallel testing on or off. Now for the most part, all you need to do to get the benefits of parallel testing is just to turn it on. But here are few tips and tricks to help you get the most out of this feature.

First, consider splitting a long-running test class into two classes. Because the test classes execute in parallel, testing will never run faster than the longest running class.

When you run tests in parallel, you may notice a situation like this, where one test class dominates the overall execution time. If you take that class and divide it up, Xcode can more evenly distribute the work of test execution between the different runners, which can shorten the overall execution time.

Now don't feel like you need to go and take all of your classes and divide them up. That shouldn't be necessary but if you notice a bottleneck like this, this is something to try. Next, put performance tests into their own bundle with parallelization disabled. This may seem counterintuitive, but performance tests are very sensitive to system activity so if you run them in parallel with each other, they'll likely fail to meet their baselines. And finally, understand which tests are not safe for parallelization.

Most tests will be fine when you run them in parallel, but if your tests access a shared system resource, like a file or a database, you may need to introduce explicit synchronization to allow them to run concurrently.

Speaking of testing tips and tricks, if you'd like to learn more about how to test your code, I'd encourage you to check out our session on Friday hosted by my colleague Stuart and Brian . You won't want to miss it. To wrap up, we started today's talk with code coverage and the new improvements to performance and accuracy.

Next, we talked about new test selection and ordering features, which allow you to control exactly which tests run, as well as the order in which they're run.

And finally, we talked about Parallel Distributed Testing, which allows you to distribute test classes between different runners to execute them in parallel.

For more information, I'd encourage you to download our slides from developer.apple.com and come see us in the labs later this afternoon.

Have a great WWDC, everyone. [ Applause ]