The Problem
You may be interested in how to wire up or sync CoreData to a remote web service. There are plenty of frameworks to do this, like RestKit, RestfulCoreData, and various defunct libraries (CoreResource, etc.) The problem with these libraries is twofold:
- They assume things about your backend (like 'It's REST!' or 'It has sane queries!' or even 'It parses!') that you know are not going to be true of whatever backend you're working with. Especially dangerous are super chatty protocols that pull down more data than you need, which you know will change halfway into the project when everyone discovers that it's slow, causing you to rewrite a bunch of fetches. Over and over again.
- They require you to run data queries against some arbitrary Cocoa API, which then multiplexes to both CoreData and the backend. This is bad because it's not portable, because if the backend API changes, your code will have to change, and because it's unclear to what extent CoreData features like faulting and proxy arrays 'just work', which you intend to rely on for performance.
Back to the future with a theme with default rounded corners of explorer's window, in os X style, made from ZEUSosX. In ancient Athenian also Ionian. The best response to change is to see how you can exploit it. Let's get ahead of the field, for once. Ancient Warfare 3. Age, high cultures, medieval ages, WWI, WWII and compare them to modern and future weapons or just recreate some historic battles.
Baldis basics plus amazing demo mac os. As a result of these items, people end up rolling their own sad little sync engine that is just a bit slightly custom for each application, highly coupled to the backend and changes as it changes, and cannot be effectively re-used. Or alternatively, they end up writing their own [MyAPI doSomeKindOfFetch] helper classes that require a lot of thinking in application code to use correctly (caching policies, etc.)
Instead of doing this, you should be using NSIncrementalStore.
NSIncrementalStore
NSIncrementalStore is perhaps the best-kept secret in iOS 5. It doesn't show up in any advertising materials, barely rates a passing mention in WWDC tech talks, and has the world's shortest programming guide, at only a few pages. It is so well hidden that I only discovered it very recently. But it's easily my favorite new iOS 5 feature.
Essentially you can now create a custom subclass of NSPersistentStore, so that instead of your NSFetchRequest hitting a local SQLite database, it runs a method you define that can do something arbitrary to return results (like make a network request). This means that fetching remote objects is now as easy as
Even cooler, we now support network faulting. So a code snippet like this:
(Of course, it doesn't have to be. Your custom subclass can fulfill the request however the heck it wants to, including from a local cache.)
So why should you be using NSIncrementalStore? One reason is because it lets you, in application-land, use maximally expressive queries, independently of how bad the backend is today. Perhaps you are working with a terrible backend that only supports 1 API call that dumps the entire database. You can still write powerful queries in your application to request highly-specific data. Then you can respond to those application requests by serving up the highly-specific data from cache.
But when the backend guys finally wake up and realize it's a bad idea and give you five different kinds of requests, you only have to make that change once. The application is still emitting highly-specific queries. You look at the query that the application wants you to serve and select the API call(s) that are most efficient to serve that query, doing whatever sort or filter that the backend doesn't handle as an in-memory step. And you only have to worry about this in one place.
And then as the backend slowly improves, you can slowly take advantage of more and more features. As a slightly absurd example, if some day they let you execute raw SQL (the horror!) directly on the backend, you can write a little SQL emitter that takes CoreData queries and emits SQL that fulfills them. The application continues to work at every step, it just gets incrementally faster as the network traffic gets more efficient.
And CoreData really is a fantastic API to use to talk to remote web services, because it lets you interact with remote objects as if they were local objects. Everybody has forgotten–this was originally the problem that Objective-C / Cocoa was designed to solve! We do not need yet another function call to retrieve a pansy flat representation of a record on a server. ObjC has had a better solution than that for 30 years!
NSIncrementalStore – the unofficial guide
As cool as NSIncrementalStore is, it is missing a lot of love from the Apple documentation writers. (Like any new API, there are undocumented bits, and some scary legitimate bugs, that nobody has gotten around to fixing yet.) The best resources on it are
- The NSIncrementalStore and NSIncrementalStoreNode class reference
- The Incremental Store Programming Guide (which is unfortunately way too short)
- A brief treatment in this WWDC talk which is illuminating
- The various resources on NSAtomicStore (class references, programming guides) are also very helpful in filling in missing details. NSAtomicStore is another type of subclassable CoreData persistent store which was introduced some time ago and is better documented.
- (and hopefully, this blog post)
Init
The first thing you have to do, of course, is override init in your NSIncrementalStore subclass.
Load metadata
When you chain to the super initializer, you will get a loadMetadata: callback. Note, and this is very important: if you fail to set the appropriate metadata here, your incremental store will be improperly initialized. This can result in strange EXC_BAD_ACCESS errors that I see people complaining about on devforums.apple.com.
You are required to set up a NSStoreTypeKey and an NSStoreUUIDKey, and if you fail to do so the incremental store will not be initialized. See the appropriate documentation in NSPersistentStoreCoordinator to learn more about these keys.
Setting up NSPersistentStoreCoordinator
Now, elsewhere in your application, you set up the NSPersistentStoreCoordinator. You tell it to use the custom NSIncrementalStore like this:
Easy as punch. Notice on line 1, we set the string to 'mystoretype', which is the same string we used in our metadata. Failing to make these strings match in my experience sometimes (but not always) causes the persistent store to fail to initialize, with an error message complaining that the strings don't match. So my advice to you would be to store that string in a common location (like in the .h file of the NSIncrementalStore) to ensure that they match.
Handling fetch requests
Now we have to actually handle the fetch requests. This is where things get tricky, because this method
- Can handle fetch requests or save (e.g. write) requests
- Can handle fetch requests that are looking for an object, fetch requests that are looking for a count, or undefined future request types
- Does not actually return any of the object's properties or attributes, which are requested on a different callback
- But is responsible for filtering or sorting on those properties
I'm going to show you just a dummy implementation to get you started, because the full scope of things you can handle in this method is extraordinary.
Note that you are responsible for handling all filters and sorts set on the fetch requests (if your backend does not). This is a lot of work, and I am considering some way to abstract this work so that there is a 'fall back' in-memory sort and filter implementation that can be re-used between NSIncrementalStore subclasses, but I have not yet thought through all the details.
Note that I am returning nil, but (oddly) not setting the error. Why is this? Because due to an undocumented bug in NSIncrementalStore (rdar://10732696), you cannot reference an error created inside an executeRequest:: call from outside the call. So if you do this:
If you want your errors to survive the end of the method, you must make a manual call to objc_retain when you create them. Yes, really. File a duplicate bug with Apple.
Faulting
Allright, so now we've returned a set of fault objects to our application. But what happens when our application tries to access the value of a property? Our NSIncrementalStore class needs a new callback:
This method handles the fault fire of our managed objects (at least for property values). A similar callback exists that fires faults for relationship accesses.
A warning about threads
Generally speaking, when you use CoreData, you are interacting with local queries. So calling managedObject.property is not terribly slow, and is something that can probably reasonably be done on the main thread.
However, when you introduce expensive remote queries into the mix, it's not so simple. Suddenly, accessing the property of an object can take a few seconds, or can potentially not be available if you've retired to your concrete-enforced nuclear bunker (or if the wind changes direction and you're using AT&T).
Since you probably don't want to do network requests on the main thread, you are going to be thinking about how to move CoreData code into the background. CoreData threading is a scary and mystical topic, and I've talked to plenty of smart people who don't really understand how it works.
Ancient Future Mac Os Catalina
The 'cliff notes' answer is that everything is threadsafe except NSManagedObjectContext (e.g. the thing you use to run queries) and NSMangedObject (e.g. your entities themselves). If you are doing cool things with threads, you need to make sure that you have a separate context for each thread and that you do not pass NSManagedObjects between threads. You can serialize an NSManagedObject to its ID and then get a new NSManagedObject from the new context using the ID.
But a better pattern for most simple applications might be to do all CoreData access on its own thread and use callbacks to retrieve the results of queries. I am considering adding some support for this in my CoreDataHelp project. For example:
I'm still playing around with the syntax. I like the first method quite a lot, but I have mixed feelings about whether wrapping a property fetch is a good idea in practice (or if I should try to rely on pre-caching the property values and only faulting to cache). It may take me a few applications to settle on a pattern.
In conclusion
NSIncrementalStore is one of the best-kept secrets in iOS 5. It has the potential to really change how we interact with remote data on iOS. Today, integrating with web services on iOS is complicated, and doing anything nontrivial requires importing three different (conflicting) JSON parsers, dealing with SOAP fail, importing laughably bad libraries like ASIHTTPRequest that everybody seems to use for everything, doing low-level NSURLConnection hacking that is (at best) very hard to do correctly, and much more. You need about fifteen tabs of documentation open on a second monitor just to get anything done, let alone how badly your application code gets polluted with backend implementation details. Not to mention that one service you're using that has taken it upon itself to produce its own poorly-written ObjC bindings that somehow has made the exact opposite design choices that you would have made for your application, and so you have to fork and hack it pretty aggressively to make it work, changes which will never make it upstream.
Now consider the alternative: we start treating the NSIncrementalStore subclass as the way to wire up remote services to Cocoa. (Either the remote service or the community starts wrapping up GitHub, Flickr, et. al. into NSIncrementalStores as the recommended way to talk to remove services.) Now you don't have to think about whether they're really using REST or only sort-of using REST, or whether their JSON really parses or which set of requests you need to make to get the data you want. All you have to do is understand the object graph they expose (e.g. read a couple of class references), write a couple of NSFetchRequest calls, and you're done. The application interface is the same for every service. This radically simplifies the complexity of writing API-consuming applications.
All the players in the iOS backend space (Parse, Kinvey, and Stackmob) are betting the farm that you would rather talk with Cocoa objects than worry about REST. And they are right. But with NSIncrementalStore, you can graft object support quite beautifully onto arbitrary, existing APIs. And not only that, but you get the power of decades of improvements to CoreData out of the box (like faulting, uniquing, a powerful query language, batched queries, undo support, and much, much more).
But I think that Parse et. al. have much to worry about. NSIncrementalStore is a complex beast, not for lesser developers, but it will gain a lot of mindshare on account of the fact that you can shoehorn existing APIs into it, something that will never be possible with cloud providers. These two forces will cancel each other out. Meanwhile, developers will be concerned about the long-term viability of businesses like Parse, or vendor lock-in, and if they have experience maintaining NSIncrementalStore code for GitHub et. al. they may in many cases consider NSIncrementalStore+REST a better candidate for new APIs than a proprietary service, in spite of its additional complexity. Plus you get decades of cool CoreData tech for free. (One area in which Parse et. al. can still win is through thread management, which is bad with CoreData as it was originally built for fast local queries, but can be fixed through appropriate and lightweight helper classes).
A call for NSIncrementalStore subclasses
So the next time you are reading Yet Another API Manual for consumption in your iOS / Mac application, consider whether the right response is to push an NSIncrementalStore subclass up to GitHub, even if it only supports a small subset of the remote API. API bindings are an area that can really benefit from collaborative, open-source development, even if you are working on an otherwise proprietary Cocoa project. I know that as I consume APIs going forward, I will be spawning lots of OSS NSIncrementalStore projects as a much better way to encapsulate interacting with remote data.
Want me to build your app / consult for your company / speak at your event? Good news! I'm an iOS developer for hire.Future Mac Computers
Like this post? Contribute to the coffee fund so I can write more like it.In the 1st degree mac os. The Seattle Asian Art Museum is set to re-open February 2020 after closing for a $55 million renovation and remodel. The re-imagined and re-envisioned Asian Art Museum now offers a thematic, rather than geographic or chronological, exploration of art from the world's largest continent. An innovative partnership between the City of Seattle's Office of Arts & Culture and the museum has brought an unprecedented focus on local and contemporary Asian Pacific Islander (API) artists in a series of free workshops, pop-up events, temporary installations, and community gatherings curated by local Corean artist Che Sehyun. Through a competitive process, Che's public art proposal titled The Future Ancient was selected to expand SAAM's opening events on the weekend of February 8 and 9, 2020, and continue throughout the summer to showcase the incredible talent of local API artists and community who value their cultural heritage in a future-forward and socially transformative way.
Ancient Future Mac Os X
The Future Ancient is an empowering model of public art, strategic community investment, and a curatorial process, which will present six months of free public programming that centers the life and perspective of local API artists. In linking the mission of the Seattle Art Museum – to connect art to life and foster creativity and community – and intersecting with Seattle's Office of Arts & Culture's values of equity, inclusion and access – The Future Ancient invests in and empowers local API artists doing the work through dollars, institutional support, and professional development alongside six months of community-lead public art engagements. The Future Ancient is a framework that seeks to expand our consciousness and appreciate our diverse cultural heritage and past legacies in future forward thinking and transformative way.