Core Data Best Practices

Good afternoon, everyone. Welcome to Core Data Best Practices. My name is Scott Perry. I work on Core Data. And we'll be joined later by my teammate, Nick Gillett.

The plan for today, first, we'll talk a bit about how the process of getting started with Core Data has evolved over time. Then, we will cover ways we can evolve our application more easily by taking advantage of extension points in the persistent container. We'll follow that with how we can evolve our model as our apps requirements change and our data grows.

Then, Nick is going to talk about a few ways our app can maintain its performance, even as it scales beyond our wildest dreams, and we'll wrap up with some good stuff about transformers, debugging, and testing. But first, let's build an app. I like to take photos, so we're going to build something that allows me to share photos with friends and get comments from them, even if it's just Nick asking how my slides are going.

Where should we keep our app's data? Well, we could keep it all online, but I usually take photos when I'm traveling, and the connection can be kind of spotty, so we should keep it locally, organized into some kind of store.

So, we have posts and comments and their instances and the relationships between them form an object graph and we've decided we need to persist these things on disk, so that's what Core Data is for.

So, we'll use that and we'll start by translating our mock here into something a store can understand, a managed object model.

We'll need fields for everything, with attributes for things like the image's data as well as the time it was posted, and we'll need relationships for posts and comments. We've also defined a need for a store, but there's a lot involved in maintaining data on disk over time.

Luckily, Core Data provides a persistent store coordinator to manage that. The coordinator can do things like compare the app's model with the store's version and automatically migrate it forward as our app evolves. Finally, managed object context provide safe, fast, and predictable access to our data, even when we're using many at the same time through features like query generations, connection pooling, and history tracking.

Setting this all up requires finding the model and loading it and deciding where to keep the store, but a lot of these error paths can't actually fail once you've shipped your app, so Core Data provides a container type that dramatically reduces the amount of boilerplate required to set up your stack, just refer to the model by name, and the persistent container will load it out of the main bundle and keep a stored in a consistent location. This persistent container type encapsulates a whole stack and includes conveniences for a shared main queue view context as well as factory methods for generating background contexts as well as performing background work. It's also designed to be easy to work with as our app grows. For example, let's say we want to factor our model layer into its own framework. We can do that by creating a new framework target in Xcode and moving our code into it. It's all super easy, but when we move our model into the new target, in the built product, targets move from the app into the new framework, which is what's supposed to happen, but now NSPersistentContainer doesn't know where to find our model anymore. This is because it only checks the main bundle by default. Why stop there? Well, searching all of the app's bundles could get really slow for a complicated app and it's not a cost you want to pay every time you spin up a stack.

How do we fix this? Well, we could resuscitate the model out of the framework bundle ourselves and use one of the container's other initializers, like one that takes an explicit managed object model, but NSPersistentContainer actually has a way for you to change which bundle it searches.

See, NSPersistentContainer knows when it's been subclassed and will use the type of the subclass as a hint when it looks for the model. All we need to do to take advantage of this is to create a subclass. It doesn't even need to have anything in it.

Then, any code setting up through the container that wants to use our model can just adopt that subclass and the persistent container will check in our frameworks bundle for our model instead. So, that's fun, but since we're going through the effort of factoring our app's resources, wouldn't it be nice if we also improved the organization of our data on disk? By default, new persistent containers come with a store description for an SQLite store with automatic migration that on iOS lives in our app's documents directory. That was great when our model code was part of the app, but we should try to keep our new frameworks files from mingling too much with the apps.

Since we already subclassed NSPersistentContainer to make finding the model easier, let's build on that to improve this.

The brute force way to change a store's location is to directly modify the URL in the persistentStoreDescription before loading the store.

Sometimes that's what you want and we could use that pattern here, but we don't have to because NSPersistentContainer calls its own default directory URL method when creating persistent store descriptions. And it's made to be overridden. In this case, we can just append a path component, but this is also a good way to set up containers for caches or other kinds of stacks that need to keep their stores in different locations, like your tasks. So, now that we've got our Core Data stock all figured out, let's have a look at our app and some of the view controllers that we've written. It looks like we've got some pretty specialized view controllers here. Here's one that shows all of my posts as well as another that shows all posts by all authors. Even the detail views are duplicated. It feels a lot like we could have written half the code. All we should really need is one view controller for displaying a list of posts and another for displaying a single post. We can accomplish this by defining good boundaries in between our view controllers in the form of interfaces that take model objects.

Each controller gets configured by its model parameters and then they can customize their views in cells based on whether they're showing my posts or someone else's. When drafting view controllers using Core Data, list views should get fetch requests and detail views should get managed objects.

View controllers also need a managed object context, either the container's view context or some other main queue context. And this pattern for generalizing view controllers with Core Data isn't just for UIs; it works really well for utility types as well.

Instead of passing Core Data types for presentation, we can pass things like URLs or serialized data into background work controllers and turn those into new and updated managed objects using a background context instead of a view context to do our work.

Adopting this kind of interface and utility type is super easy since we own the initializer, so we can just require the parameters to create the controller. But how do we get our boundary variables into our view controllers? Well, if we're using segues, we can override the prepare method and get a reference to the destinationViewController and then configure it there. If we're using storyboards or nibs, then we already have code that has to cons up a destinationViewController, so all we need to do is set the properties before presentation. And, if we're driving stick, we can just write an initializer that explicitly defines the boundary conditions, just like we do with our utility types. OK. So, now we've got a fetch request and a context for our view controller, but before we smash them together to get our results, we should configure the fetch request a little bit more to make sure that our controller will have great performance.

Sometimes it makes sense to set a fetch limit, but in the case of our list view, batching makes more sense because we want to show all the data and we know exactly how many cells our view controller can fit on the screen at once. In general, at least one of these options should always be set for fetch requests that might return an unbounded number of results. So, at this point, we could turn our fetch request into objects and populate a list view with the results, but what if we want to keep the UI up to date with changes as they happen? Core Data has us covered here as well with the fetched results controller. Available on all platforms since Sierra, adopting the fetched results controller just requires writing an adaptor between the delegate protocol and the view it's driving. And to create one, all we need is a fetch request and a context.

The fetched results controller even supports driving more advanced list view concepts such as sections.

If we wanted to group posts into sections by the day they were posted, we could do it by extending the post type that's generated by Xcode with the computed property and passing the name of it to the fetched results controller's initializer.

This works well, but what if we have a view controller more complicated than just a list of our objects? What if we want to show something like a chart of the posts per day on our app? Well, the first thing we should do is not underestimate the power of fetched requests. I'm just one person, so in the last month, I haven't managed to post more than 40 pictures per day. Over the course of 30 days, that's still a fairly reasonable amount of data to pull out of the store at once. If the day property that we defined earlier was actually part of the entity in the model, then we could write a fetch request that counts up the number of posts grouped by the day they were posted. There's three parts to this request. The first one is just setting the range. We want the last 30 days of data. Next, we want to group together all results whose day attributes share the same value. Since we're now fetching aggregates instead of individual objects, we have to change the result type to something more sensible as well, in this case, a dictionary.

Finally, we define an expression that represents the number of objects in each group and tell the fetch requests to return that count along with the day it represents.

This fetch request returns 30 results, each of which is one point on our chart.

If you're into databases, this is the SQLite query that Core Data generates from that fetch request. It's exactly what you do if you're writing the query yourself. Core Data understands how to convert many expression functions into optimal database queries. A group by query can use aggregate functions such as average and sum and scalar queries, like a normal fetch request, can use scalar math and date functions, like abs for the absolute value and now for the current time. If you want to know more about what you can do with NSExpression, check out the documentation for its list of functions. Many of them are supported by fetch requests in Core Data.

OK. So, fetch requests can accomplish a lot through the use of expressions, but SQLite still reads every one of our posts through memory when computing the counts for our graph here. That works fine for charts showing the amount of posts generated by one human in a month, but what if we want to chart something bigger? What if we want to show a whole year or what if our little app starts handling orders of magnitude more data? Now the fetch request would be counting at least 50,000 posts one by one just to show 30 data points and that's not going to be fast enough. The mismatch between our views and our model has gotten to the point where we need to start doing some denormalization.

Denormalization is when we add redundant copies of data or metadata to improve read performance at the expense of some additional bookkeeping. Database indexes are a good example of this.

Adding count metadata to our store is exactly the kind of compromise we need to get our chart's performance again. So, let's look at how our model can group posts into counts by day. We'll need a new entity with two attributes, plus a bit of extra maintenance to keep them accurate. Grouping by day improves our fetch request so much that it guarantees good performance for charts covering years of data, so we only have to create this one level of denormalization and the fetch request that we passed to the chart view controller? It's super simple.

It's really not that much different than the fetch request that we'd pass off to any other list view, which is actually kind of sort of what a chart view is if you squint hard enough.

But what about that extra maintenance? We've got to increment the count whenever a post gets published and decrement it whenever a post is removed. We could do this in the methods that change the post object's relevant state, but a more foolproof solution is to update our counts in response to the context saving. We could just register for the managed object contextWillSave notification with a function that goes through all of the posts that have been inserted, incrementing the count for each day that's relevant, and another loop going through all of the deleted objects, decrementing the count for each day. And that affects the state of the context before it commits to the database, so it all winds up happening in one transaction, so this scales really well, which is useful because my teammate Nick Gillett is here to talk about how Core Data can help us as our little app scales beyond our wildest dreams. Nick. Thanks, Scott. So, as Scott said, as your applications grow, they get more and more complex and at Core Data it's really important to us that your applications do grow. In fact, we want this. That's the whole reason we exist is to help you manage this growth, make it efficient for you to work with, and help you give more value to your customers. But this happens in ways that are very specific to your application. It's also highly aligned with your customer experience or the way that you want them to experience your application. Unfortunately, like all complex systems, as it grows and becomes more complex, it also tends toward chaos. So, today we're going to talk about ways that Core Data can help you manage this chaos and give it some structure. We'll talk about building predictable behaviors and helping you build tunable containers that you can align with your experience metrics.

What does that mean? Well, when we think of metrics, there's a couple of different ways that we can think about them. The first is in alignment with our customers. And usually we define these as things that they experience, like having a consistent user interface or a responsive scroll view, and also as delight. But those are really hard for us to capture as engineers. So, we translate them into engineering metrics, things like peak memory consumption, how much battery drains during a given task or how much CPU time is burned during a given task. And, finally, how much IO we do during a given task.

To make this a little more concrete, we'll use this application. Some of you may remember the history demo application that was introduced last year at WWDC and I've modified it for the purposes of this talk. There are a few actions here that a customer can take while using our application. The first is they can add a single post to our database by hitting the + button.

They can also download any pending data from the server by tapping Download.

And, finally, for anything that hasn't yet been uploaded to a server, they can tap Post All.

Now, this application has a fairly small set of interactions that a customer can take, and yet, as these happen concurrently, it tends towards chaos. So, we can see that even with this small set of actions, things that go on concurrently could cause a number of different state changes in the application and the worst thing for us is to end up with a user experience that looks like this. This notion of partial completeness doesn't make sense to our customers. In fact, it doesn't make sense to us either.

Core Data is here to help with that with query generations. Query generations were introduced in 2016 in our What's New in Core Data session. So, if you're not yet familiar with them, I highly recommend that you check out that session for more information about how they work. What you do need to know is that they require wall journal mode and only work with SQLite.

The goal of query generations is to isolate your managed object contexts from competing work. This could be rights to the background or actions that the user is taking that you're not yet ready to manifest in a given context.

Query generations provide a consistent, durable view of the database that will return the same results for fetches regardless of what other contexts are writing to the database at a given time.

The best part is we can adopt them in one line of code.

This is a typical change for reloading a table view. We would just have to insert a call to NSManagedObjectContext setQueryGenerationFrom token with the current query generation.

And when it comes time to update them, we can update them as we normally do by using NSMangedObjectContextDidSave notification. And this allows us to manifest changes to the application's data in the UI at the right time.

But what if the data that we're writing isn't related to the UI, such as downloading some comments that Scott mentioned earlier? In this case, we don't want that data to manifest in the user interface or cause changes to it because none of the change will be visible to the user. So, we can actually filter out these updates by using history tracking. Persistent history tracking was new in iOS 11 and macOS 10.13. We introduced it in our session, What's New in Core Data, last year at WWDC, and for more information about how it works and what the underlying features are of it, you can use that session as a reference.

Persistent history tracking is a great way to get a persistent record of each transaction that connects to the database and this is useful to us for a couple of different reasons. For the purposes of this talk, though, we'll be considering NSPersistentHistoryChange, which gives us a changedObjectID and a set of updatedProperties. And NSPersistentHistoryTransaction which gives us a set of changes and an objectIDNotification.

So, let's consider the following set of changes. As you can see, these are posts that are being inserted to our database and when this happens, given our table view, we would want to refresh the UI, which we can do by using the objectIDNotification. These are analogous to NSManageObjectContextDidSave notifications and can be merged in using the same API.

But if we downloaded a list of comments that we don't want to manifest in a user update for, we can filter them. Using this small amount of code, we can filter out the changes from a given transaction to decide if any of them were relevant to the post entity and in that way we won't refresh the UI and cause an unnecessary blip or stutter to the user.

But as you can see here, we're actually only using a small amount of the post content. In fact, we're only using two properties, the image and the title.

And so we can do better than just filtering out by entity. We can actually filter out by updated properties using the history changes and in this way we can create highly-targeted updates to our user experience that align with changes that will be visible to them. But Core Data can also help you support new interactions for your users. As your data becomes more complex and grows in scale, some editing operations can get more expensive. For example, consider a simple photos browser. Typically, when our applications grow, we want to -- we want to introduce new functionality that makes it easier to perform repetitive tasks, such as multiple selection. And Core Data can support this by using batch operations.

In fact, in just a couple lines of code we can mark an entire set of photos as a favorite or not. And in just one line of code, we can purge or delete a set of records from the database using a batch delete operation. And these operations scale in ways that aren't possible by faulting the objects into memory. For example, during a delete, a traditional delete by calling NSManagedObject.delete will grow with the size of the records in the database. And as you delete objects and their memory gets faulted into the context, this gets more and more expensive the larger your database gets. But with batch operations, we can perform the same mutations in just a fraction of the memory. And this has the curve that we want as data increases, where the larger the data set is, the less memory we use, using up to about 7% of the memory of a traditional delete at 10 million rows.

So, this is a very powerful way to save resources on your customer's device. But one of the traditional problems with batch operations is that they were difficult to work with because they don't generate save notifications. That's where history tracking comes back in. With persistent history tracking, we can fetch out the transactions from the database that occurred as part of the batch delete or update and we can use the objectIDNotification method to generate a notification that works like a save notification. In this way, the fetched results controller or any other context in your application can update to those notifications incrementally. And so those are some ways you can manage growing data with Core Data, but how about your workflow itself? What can Core Data do for you as a developer and engineer to make it easier to build and test your applications? The first thing is that we can help future you today.

As you may know, NSKeyedArchiver is changing. We're adopting secure coding across the entire platform and the KeyedArchiver API has changed significantly this year in support of that.

For Core Data, this means that value transformers are changing, so if you have a transformable property in your managed object model and you're not sending a value transformer today, you used to be getting NSKeyedUnarchive FromDataTransformer as your default value transformer. In the future, you'll be getting NSSecureUnarchive FromDataTransformer and this implements secure coding under the covers and you should adopt it today. There was a great talk this morning about exactly this topic called the Data You Can Trust and I highly recommend that you view it to get more information about secure coding and how to make your applications more resilient.

You can specify this in the model editor with a transformable property by the value transformer name field.

And today, we want you to implement this on your own. This will become the default in a future release and in a future release Xcode will also start emitting a warning for anyone that's using the default value transformer name.

If you're building your models in code, you can set it by using the valueTransformerName property on NSAttribute description.

This should be transparent for you if you're not encoding custom class types. So, for plist types, this is a no op. You simply change the value transformer name and you'll get the new secure coding behavior. However, if you are implementing custom classes, those classes need to adopt secure coding and you can come see us in the labs for help with that. But we can help you more. At Core Data, we've spent time building in new debugging tools over the years that can help you understand what's going on under the stack. So, this is a picture of our preferred default scheme configuration. We have a couple of process arguments that we have that can help you get more debugging information about SQLite, but the big one you should always run with is com.apple.Core Data.ConcurrencyDebug. And this will catch any queue exceptions in your applications, so areas that you may be transferring objects between main and background queue contexts or areas that you may not be obeying a managed object's actual context. SQLite also has a number of interesting environment variables, so their threading and file assertions are a great way to ensure you have correctness in your application around their API as well as the file system. And auto tracing is a great way for you to see what's going on under the covers, an additional tour on debug logging. com.apple.Core Data.SQLDebug has four levels. The first level is the most interesting and the least of a performance hit. The fourth level is the most verbose, but does cause significant performance hit when you run with it. When you enable SQL debugging and our multi-threading assertions, you'll see a couple of logs in the console and these are the indication that the assertions are enabled and running correctly.

With our SQL debugging on, you'll be able to see things like select statements for our fetch requests as well as how long they took. And, if you're set to level four, you'll even get explain, which will show you the query plan for a given select statement. And here we can see that our table view is selected via table scan and then using a temporary B-tree in memory for the order by, which is on the timestamp. This is a potential performance problem and as you're running your application you can use messages like this to see where you may be doing more work than you need to.

So, how would we fix this? Well, turns out SQLite 3 can actually tell us. If we open a database and hand it the select query from our SQL logs, we can enable a mode called Expert, which will analyze the query and give us the ideal solution of optimizing it by creating a covering index.

And we can do this in the model editor by adding a fetch index to our post entity. Here I've configured it to run on the timestamp and fetch them out in descending order because we're showing the most recent posts at the top of the table view.

When we run the application again, we see the same select logs.

Except that this time we see that the select query hits the covering index during the query. Explain shows us that the query will use the covering index for its order by. Core Data supports many types of indexing, including compound indexes using R-trees. And these are great for creating any kind of query or optimizing a query that uses a bounding box in its select statement. This is most commonly done with locations and we can set this up by adding another index to our post entity, which works in the latitude and longitude property that I added for the purposes of this slide.

We change the query type in this box by selecting R-tree. And then we can set up our predicate on the fetch request to say get all of the posts that happen inside of continental China.

This predicate is a little more advanced because it uses functions inside the actual select statement to hit the index that we created in the managed object model.

When we run our application without this predicate and without this index, we see the same results that we saw before where we're hitting only the timestamp index. But when we run it with our new index and predicate, we see that SQLite is using the index to generate faster results for both of the between statements.

Unfortunately, because our timestamp index doesn't have any bounding predicates on it, SQLite can't use it for the sort. So, the optimization that we've chosen here is to use a compound index to first filter out the result set to a smaller set of objects and then we'll do an in-memory B-tree sort for the order by. As you can see, this index increases the performance of our fetch by about 25%.

In this case, my performance test was run over a size of about 100,000 rows and we saw around 130 milliseconds of improvement for just the fetch.

Which brings me to my next topic of testing with Core Data.

As you may know, we really like tests. Tests are awesome. And, at Core Data, we use them internally for both correctness as well as learning. They're a great way to learn about the functionality of Core Data and how our API behaves under a given set of conditions.

They're also a great way to verify your assumptions about how Core Data works and how it's going to help your customers have a better experience with your application. As you saw in the previous example, we can verify that our R-tree index actually does give us a performance benefit even though it's using an in-memory B-tree sort.

They also capture your product requirements, and this is really important to us at Core Data because it helps us communicate around your expectations. With tests, we can see what you're doing in code and how you expect those lines of code to behave for your customers.

There are some important things that you can set up to make this easy for yourself, such as a base class that generates a persistent container.

This base class on the screen happens to use a file URL of /dev/null for the persistent store and this is a great way of making tests that operate on a small set of managed objects run very, very quickly because they'll run entirely in memory. When you do this, SQLite materializes an in-memory store for you that can be very efficient, but because it's in memory, if you have a lot of data, this will cause a lot of memory growth in your test suite.

You should have at least one test, though, that actually materializes your store file on disk. And this is because if you can't open your store for your test suite, it's highly likely that your customer can't either.

If your persistent container is in the application delegate, you can have a test base class that grabs the container out and writes directly to that store. But I must caution you to take care when you do this, because that means that you're writing to the store file that's in use by the application, so if you run your test on a personal device, you'll see the effects of the unit test when you open your application the next time. What if I told you I could insert 100,000 records in just seven lines of code? I'm cheating a little bit. I was going to leave this as an exercise to the reader, but this type of scaffolding is a great way to help you build a test suite that evaluates your invariance around your data.

By building these methods ahead of time, as your data changes or you become aware of new use cases for your application, you can iterate on these to build new edge cases, build new structures for your object graph, or evaluate the behavior of certain functionality under the covers, such as performance. This is the unit test scaffold that I used to build a performance test for the R-tree query. In just a handful of lines of code, we get high confidence on the performance of our fetch.

And these types of tests are very informative when you're trying to evaluate tradeoffs between different features and functionality in Core Data. These three lines of code generate a new managed object context and container for us for our test to use. Now, this is important primarily because the setup and teardown logic in tests can sometimes affect their performance. So, you'll need to take care to analyze whether or not you're actually testing the teardown performance or the setup performance or the actual runtime performance of the queries you're evaluating.

And after you've run these tests, you can file good bugs.

We love bugs, especially from you guys because we're building a product to help you build your applications. But, bug reports without tests or without a sample application are very hard for us to communicate around because, as I mentioned earlier, they don't capture your product requirements and expectations in the same way that well-structured tests do. In fact, in an application attached to our radar that has a test suite or even just a bare sample application with some UI that explains your constraints and concerns to us, we can get back to you much more rapidly about what's going on and what you should do about it. They also help us verify the correctness of our fixes later. So, if you're going to file a bug, please take some time and write a test for us.

That's all I have today. Come see us at lab tomorrow. We are here from 1:30 on in Technology Lab 7 and I highly recommend that you check out Testing Tips and Tricks tomorrow in Hall 2 at 3:20.

Thanks. [ Applause ]