Where to find IOKit headers for iOS

Sometimes when compiling more sophisticated iOS projects you need the IOKit headers which were removed from the iOS SDK a few versions ago. A simple workaround is to copy them from the Mac OS SDK to the iOS SDK with these commands:

sudo cp -r /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/IOKit.framework/Versions/A/Headers /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/System/Library/Frameworks/IOKit.framework

Sometimes you also need OSTypes.h from libkern:

sudo cp /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/libkern/OSTypes.h /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/libkern

If you still have build errors check you are building for device and not simulator.

Default behaviour of UISplitViewController collapseSecondaryViewController

The documentation for UISplitViewControllerDelegate collapseSecondaryViewController says:

When you return NO, the split view controller calls the collapseSecondaryViewController:forSplitViewController: method of the primary view controller, giving it a chance to do something with the secondary view controller’s content. Most view controllers do nothing by default but the UINavigationControllerclass responds by pushing the secondary view controller onto its navigation stack.

Similarily, for separateSecondaryViewControllerFromPrimaryViewController it says:

When you return nil from this method, the split view controller calls the primary view controller’s separateSecondaryViewControllerForSplitViewController: method, giving it a chance to designate an appropriate secondary view controller. Most view controllers do nothing by default but the UINavigationControllerclass responds by popping and returning the view controller from the top of its navigation stack.

I thought it might be interesting to try to implement this magic behaviour to help understand what is going on and thus provide a starting point for customisation.

- (BOOL)splitViewController:(UISplitViewController *)splitViewController collapseSecondaryViewController:(UIViewController *)secondaryViewController ontoPrimaryViewController:(UIViewController *)primaryViewController {
    
    if ([secondaryViewController isKindOfClass:[UINavigationController class]] && [[(UINavigationController *)secondaryViewController topViewController] isKindOfClass:[DetailViewController class]] && ([(DetailViewController *)[(UINavigationController *)secondaryViewController topViewController] detailItem] == nil)) {
        // Return YES to indicate that we have handled the collapse by doing nothing; the secondary controller will be discarded.
        return YES;
    } else {
        
        // push the secondary view controller onto the primary's navigation stack
        UINavigationController *nav = (UINavigationController *)primaryViewController;
        [nav pushViewController:secondaryViewController animated:NO];
        return YES;

    }
}

- (nullable UIViewController *)splitViewController:(UISplitViewController *)splitViewController separateSecondaryViewControllerFromPrimaryViewController:(UIViewController *)primaryViewController{

    // respond by popping and returning the view controller from the top of its navigation stack
    UINavigationController *nav = (UINavigationController *)primaryViewController;
    return [nav popViewControllerAnimated:NO];

}

It’s strange it is possible to push the secondary navigation controller , since usually that exceptions with “Pushing a navigation controller is not supported”. Turns out within the push method it checks a private property _allowNestedNavigationControllers to allow it to pass in this case, and it must have been set by the split view controller at some point. These kind of tricks are really annoying because gives inconsistent behaviour and thus lowers developer confidence in the APIs.
It’s interesting to call: [(UINavigationController *)secondaryViewController setViewControllers:nil]; before pushing because it throws an exception that proves that nested navigation controllers are being used: Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: 'Cannot display a nested UINavigationController with zero viewControllers

I find using the plus iPhone simulator useful for testing the split controller because in portrait it is compact width but rotating to landscape it moves to regular width.

CloudPhotos – Apple CloudKit Sample

CloudPhotos : Using CloudKit with iOS is an interesting Apple sample project, last updated a year ago, I thought I would blog about some observations I had.

The table view controller uses an array of CKRecords for its data. I sort of remember Apple saying not to use CKRecord as your model class, but all I could find was this in the docs:

Customizing Records
The CKRecord class does not support any special customizations and should not be subclassed. Use this class as-is to manage data coming from or going to the server.

This doesn’t exactly say out-right don’t use as the model, but if they say don’t subclass it then to add your methods I suppose you might try to use a category. The thing is the CKRecord contains useful logic for change tracking, e.g. when modifying a property it stores the key in changedKeys. Then when you come to use a CKModifyRecordsOperation if you use the savePolicy save changed keys then it optimises by only sending those keys that were changed (I think). I suppose the issue here is this is a lot of useful behaviour you wouldn’t want to re-implement yourself in your own model class, so you do in fact want to be using CKRecord as your model class for things that will be sent back to the server. So why Apple say not to subclass it is a bit of a mystery. Anyway once you start to implement a local cache, or would like to make use of core data fetch controller and sorting, you likely will be using NSManagedObjects as your model classes. The CKRecord encodeSystemFields does not include the changedKeys array, I suppose what you can do is track the changed keys yourself on the managed object, and then when you come to transmit the record only set the keys that you want to be sent, and then CKRecord probably won’t mind it is missing the other information.

APLCloudManager is actually a controller, and is init from the app delegate. It sort of the MVC-N design (amazing video well worth the watch), where all network code is inside one controller and not in the view controllers. I say sort of because you are supposed to only manipulate the model inside here, and then model should notify the view controller new data is available, in Apples case their model is in their view controller and is updated via the completionHandler. It’s funny because in the video he says some people choose controller (who disagree controller should only be used in view controllers), and some use manager and right here is a manager example! So in this class we see all the methods for doing cloud network calls. At first I thought the saveRecord was strange because it looks like a direct wrapper around CKDatabase’s saveRecord. However it also has a dispatch back to main queue with in. This is cool because it means the UI view controller callbacks don’t need to worry at all about threading! Perhaps a good design tip to make use of. Maybe they should have at least named it savePhotoRecord though? That’s the only thing its used for. I suppose keeping it generic is ok, given the record itself is what contains the type. Actually one of the weirder things about CloudKit is in one batch you can save records of all different types in one batch, whereas CRUD API consumers would usually be saving one record type to each end point which would need multiple requests.

Again on this manager class, all the methods are void returns, meaning unfortunately there is no cancellation capability. I think Apple should have included cancellation on at least one of them to show people how it is done. Basically I think the method could return the NSOperation (wouldn’t work with convenience methods, would always need full operations) or a protocol (to hide that scary class from UI devs), then that has the cancel method. So for example if trying to refresh a table, then the user moves away, in the view will disappear the request could be cancelled.

The subscribe method in the manager is a particularly nasty one, no completion handler on it. It’s called from the main view controller’s viewDidLoad so if it fails then the user carries on none-the-wiser. Inside the method it does look for a not authenticated error, in which case it keeps retrying every 3 seconds. At this point I noticed the operation’s qualityOfService was not set to user interactive. This means it operates in the NSURLSession discretionary mode where it does fail with network errors and just keeps retrying on its own. I just wonder is this enough to mean that you don’t need any error handling at all? Hence why they didn’t even bother with a completion handler? Will need to look more into that. I think tho at least the user could have none not to leave the app until the subscription had been set up. I was thinking this kind of initialisation feature of checking for account/creating zones/creating subscriptions might be best done in an on-boarding UI screen. You know the kind that is like a welcome screen where there is some info, maybe a spinner while things are being set up then a button? The Apple News app has this on first launch.

In the APLMainTableViewController I’m dissapointed that upon a pull-to-refresh it calls loadPhotos which does a full download of all the photos again, and then it does a full table reload. Ideally we would want to download the changes and then insert/update records. Then we realise they are using the public database which doesn’t support the fetch changes feature, so now we know why they are doing a full download. They still however could have done the table delta updates, but maybe that was too much code for this intro to CloudKit.

Finally I’d like to make a general comment on this old project and why perhaps it is no longer a good beginner sample. CloudKit is all about privacy, so sharing photos to a public database does seem to conflict with that just a bit. In iOS 10 they added CKShare and secure sharing capabilities between friends, it would be great to see a photo sharing sample that uses that instead. I think most people that are struggling with CloudKit are trying to do caching or synchronisation, this sample doesn’t touch on that at all. Maybe there are other Apple CloudKit samples I haven’t discovered yet, but on my wish list would definitely be a proper sharing one, and a sync one (with silent notifications).

CloudKit Sync Nightmare

So after previous post on analysing the web requests, where we found out the classes involved and sort of when they should be used, we are still clueless as to how to structure our app to enable CloudKit syncing. We know we shouldn’t subclass fetch operation but then how should we gather our information to pass to the operation? Should we create our own operation that then start the fetch operation within? Or use queues, or dependencies? In general should we use our own queues or always use the container database’s built in queue? How should we structure our retry logic? Does NSURLSession’s discretionary mode handle CloudKit retry or just network failure retry? Should be be using longlife operations for sync? Should we be queuing or coalescing fetches between multiple pushes and between anything background and anything user like pull-to-refresh? So many unanswered questions. What a nightmare, Apple has been absolutely useless in their code samples at the WWDC talk (e.g. 16m 48s not useful example and didn’t even bother to demonstrate record sync). And what the hell do they mean by the requests sent last 24 hours / 7 days stuff? I think I need to class-dump the headers to figure out how Apple are doing this. So turns out the app uses an embedded framework for the CloudKit code, here is the dump of NewsCore on my Github, let’s take a look.

First of all we notice 336 files. 336 files just for the networking and caching, not the UI. Unbelievable, but lets carry on anyway.

A lot of the CloudKit related files are prefixed with FCCK, 27 of them! And there are many more like controllers etc. that use these objects. Maybe half the files have a reference to some aspect of CloudKit one way or another.

FCAppConfiguration looks like a possible entry point for this framework, it contains a FCCloudContext which looks like the heart of the CloudKit stuff, with many controllers, centres, managers and queues. Looks like it stores the controllers for syncing the reading list, FCReadingList which is a subclass of FCPersonalizationData : FCPrivateZoneController.

From FCModifyRecordsCommand we notice the coalesceWithCommand method so they are coalescing commands, this might eventually prevent duplicate network requests for the same thing, but doing it at the command level rather than the network level.

The operation subclass hierarchy is as follows FCCKFetchRecordChangesOperation : FCCKOperation : FCOperation : NSOperation

FCOperation isAsynchronous (no other reason for implementing that method) and childOperations suggests they are implementing operation grouping not with sub-queues but instead with dependencies between operations and tracking the group using an array, which would allow the group to be cancelled after being added to more global queue.

FCCKFetchRecordChangesOperation is like a wrapper around CKFetchRecordChangesOperation. Since that class isn’t an ivar anywhere it suggests its declared locally, which means they didn’t need to deal with self retain cycle block bugs. It has properties for everything needed to perform a fetch changes operation including: database, zone, desiredKeys, allChangedRecordsByID and allDeletedRecordIDs. It even has _continueFetchingRecords and resetForRetry methods which makes it look like the operation is being retried from within this operation. Is it added to the database queue and then addChildOperations from FCOperation though?

FCCancelHandler is an interesting class that likely is for wrapping the operation object into something that is safe for a UI developer to work with. It doesn’t appear to be used for CloudKit operations though.

FCAsyncSerialQueue shows they are using serial queues however it isn’t obvious if they are implemented via addDependency on last or with maxConcurrentOperations = 1. It is used in the FCCKRecordZoneManager.

FCPushNotificationCenter is the place where push notifications are managed.

They have added some convenience methods via FCCKDatabase+Additions.h which have completion handlers but no cancellation object returned.

Like we saw in the flat binary filenames used for caching modifications to be uploaded they are using the word command in the class names, e.g. FCModifyReadingListCommand and FCCommandQueue looks to be for managing the binary file with the loadFromDisk and serializeCommands methods.

FCCKTestDatabase makes me think they are taking advantage of dependency injection for testing purposes, where they can pass in a mocked cloud database for testing instead of doing real network requests.

To be honest I wasn’t expecting this complicated an app, this looks like at least a year of coding, maybe even more. Remember the app only syncs a few record types, reading list (bookmarks), history (read articles) and personalisation data like favourite feeds. Implementing syncing is going to be no easy task at all; no wonder we are all clueless after a few hours of WWDC talks! Hopefully some of my research has helped you though.

Sorry I didn’t link every class name, WordPress’s post editor stopped letting me set links for some reason.

Investigating CloudKit Sync in the Apple News App

Introduction

Syncing data across a user’s own devices in a privacy-sensitive way is one of the flagship features of CloudKit. Unfortunately many agree “it isn’t as documented as it might be”, e.g. there is no sync sample from Apple, the WWDC 2014 talk and WWDC 2015 talk didn’t quite get to the topic although did offer some vague hints, in WWDC 2016 talk the presenters gave lots of unfinished sentences possibly due to lack of time, e.g. <sic>you have to do it this way “for a variety of reasons.” Looking at the CloudKit framework headers none of the classes have sync in their name, or even have the word “sync” mentioned. Slightly better is that more recently the class doc for CKFetchRecordChangesOperation has been updated to mention this is the one to use for synchronising, however having information spread across multiple places is really hard work. Furthermore, fully understanding CloudKit behaviour requires knowledge of other things like NSOperation quality of service and NSURLSession discretionary. To add to the confusion, the open-source projects out there are all attempting to achieve sync in different ways, e.g. some are attempting to sync with the public database which is missing the required features, or they might use a periodic sync rather than realtime which CloudKit was designed for using push notifications, and zones are a really stumbling block. These common mistakes is clearly a result of lack of good samples and documentation, and in particular lack of some absolutely vital information, for example the CKFetchRecordZoneChangesOperation. I happened come across a Stack Overflow answer by a CloudKit engineer who shared that not all the changes are returned, they are coalesced to remove unnecessary ones, e.g. if a record was added and then deleted since the previous request (tracked by a token) it isn’t included. This was quite eye-opening because it shows the server is a lot smarter than expected, and knowing this now yes this could be used for an efficient sync. What also helped me is, as we see later, they do hit a HTTP endpoint that is named sync, which helped confirm this must be the right path. I think Apple could have put the words sync in the class name or at the very least put in the header, use this class for syncing!

In a situation like this where there is much ambiguity it is useful to look at how Apple do things, for some ground truth, and they are using CloudKit sync in the News App so lets take an in-depth look at that and using two devices for testing, an iPod Touch and iPhone 6s both on 10.1.1.

News Article Download

When the app starts up it shows news articles, since all users can see these articles we would expect them to be in the public database. We will use a web proxy to analyse the requests, this won’t give us exact detail of how the framework classes are being configured, but it will give us an idea of the general algorithm. We will be using the 6s to monitor the requests.

As we can see it performs a query to the container com.apple.news.public which is the public database. The record contains and articleID, title, thumbnail, contentURL etc. so the full article is downloaded in separate requests that go directly to the news organisations own servers. The thumbnail is a URL to an image on the icloud-content server you can see in the screenshot. The query used contains minOrder and maxOrder which are integers so would suggest they are doing a query to get new articles, this isn’t a sync its usually called a delta download, i.e. only downloads the new information. The articles have an order field like an auto-inc, or sequence number, newer articles have a higher number, which is better than using timestamps where 2 articles might have the same time. This is possible because only Apple is the one inserting records. This kind of delta download is a great bandwidth saver however it only allows only new articles to be downloaded which has a limitation you can see in the below screenshots.

 

As you can see the first article had its title changed between 4h ago and 13h ago. The limitation of their design is it doesn’t allow old articles to be updated, which doesn’t fit well with the news industry where headlines can be changed frequently as a result of errors or updated information. This would require existing downloaded articles to be updated, perhaps even deleted. This isn’t possible with the public database because it doesn’t have the required features to support a full sync. The alternative design would be to clear the cache and re-download all the articles every time, which would ensure the user is seeing the latest list, however that may have higher bandwidth requirements. Apple must have run the numbers for their number of records and data involved and decided an append only delta download was the way to go. There is another interesting usability feature here, as you can see we are on the history list, if a user is browsing the list to find a previously read article it certainly would make it harder to find if the title was changed. Now it becomes a very interesting problem, because you have a trade off between what is technically optimal with what is best for users.

Bookmark & Reading List Sync

Now that we have covered how articles are downloaded lets now look at the features we are really interested in, how it performs the sync between devices. The News app has two features that are synced, bookmarks and history. The history view has already been shown in the above screenshots. An item is added to the history after an article is viewed and the user has scrolled down a bit, or maybe spends some time in the article, or perhaps a combination of both. Articles can be bookmarked when viewing them, by tapping the bookmark icon on the bottom right. Now the really interesting part, if you have two devices side-by-side these two sets of data are updated almost instantly (~5 seconds) when changes happen. For example bookmark an article on one device and it appears in the bookmark list on the other device, un-bookmark the article and it disappears from the other devices list. So there we have the feature we are looking for, a full sync between devices so lets see how it is implemented. We will tackle the push part later, we’ll focus on the News App’s requests just now though.

So we have the 6s connected to the proxy and open at the saved articles page. On the iPod touch we open an article and bookmark it. 5 seconds later this happens:

As we can see in the first request it hits an end point called “zone/sync” to a container called com.apple.news.private. So now we know they are using both public and private databases for this app, which is interesting to me because obviously articles need to be referenced, and we were told many times CKReference doesn’t work across zones or databases, all they’ve done here is simply make the articleID a string field rather than a reference, I suppose they aren’t bothered about referential integrity. Next fortunately for us a familiar looking class name is included in the request, looks like a CKFetchDatabaseChangesOperation (the one in the log has prefix CKD which is because the actual request goes through the CloudKit daemon, so its like an RPC or remote class). On the response tab (not shown in screenshot) we see ReadingList and ReadingHistory which are the name of the two zones that have changed. Lets take a look at what happens next:

These next two requests are to the endpoint “record/sync” again to the private container and again we can see the class looks like the familiar CKFetchRecordZoneChangesOperation (Note. pre iOS 10 the class was CKFetchRecordChangesOperation). This first sync request contains the zone name ReadingList and as expected the second contains ReadingHistory. From more testing we see that the record/sync is only requested if the zone name is contained in the zone/sync.

Background Sync

Another feature is these lists are already up-to-date when the app is re-opened. If the app is killed and restarted then it already has the previous data which sparks my interest to see how they are achieving caching, but for now we will focus on what technique they are using for updating e.g. update on coming to foreground, background fetch or push notification. So to find out we can test this with the 6s connected to the proxy just on the homescreen, and then on the iPod Touch using news to read and bookmark an article. We’ll use the proxy and also the new Sierra Console so we can gain an insight into what the 6s is doing.

A push notification! And it has the content-available flag this shows they are using silent push, implemented using a CKNotificationInfo with shouldSendContentAvailable set. We also also see the private container and the zid which is the short version of Zone Name. Apple shorten the json key names in pushes because packet size is limited. Because the push contains the zoneID this would suggest they are using CKRecordZoneSubscription. Now lets see what the proxy logged:

Looks like the exact same requests as when the app is in the foreground. Finding out what zones changed and then what records within them changed. In fact the same push is used when the app is in the foreground, so now we know how the info is kept up-to-date. I don’t know about you but what crosses my mind is if multiple pushes are received do they coalesce the fetch changes requests? I might investigate that later on.

Pull-to-Refresh

Next, I noticed there is also a pull-to-refresh enabled on the bookmark and history table views. This was perhaps implemented as a fall-back in case for some reason the push notification doesn’t arrive so it allows the user to force a refresh of new data. Or maybe they are using the feature to clear the cache and re-download all the bookmarks to clear up any inconsistencies? Lets do a pull-to-refresh on the 6s and see what the proxy shows:

We see only a CKFetchRecordZoneChanges this time, no record download. Scrolling down the request shows it is for the ReadingList zone name. Similarly if we pull-to-refresh on the History tab we see the same request but for the ReadingHistory zone name. This proves they are only using this feature to do a sync, so as a replacement for a missing push notification, rather than a full clear-cache and re-download everything.

Caching

Now it wouldn’t be a proper sync without caching, this allows the app to be killed and restarted and still show the previous info. So lets see how they achieved that. To that we will use a jailbroken iPod Touch on iOS 9.3.3 so there is a chance that the caching has changed on iOS 10 but hopefully this is still interesting. We will connect over SSH and browse the file system to find where the News app stores its data.

The data is in /private/var/mobile/Container/Data/Application/ContainerID which was found by a process of elimination where first the mobile user’s Library folder was browsed and when nothing was found there I looked to the containers. I think this is a relatively new concept of storing platform (or built-in) app data in the container folder. Since container folders have a UUID it can be tricky to find the right one, how I achieve it is modify something in the app, like save an article to the reading list, and then sort all the containers by date. So in the screenshot above we have found the folder and we can see the private data in a folder, where i have highlighted the reading-list file, there is also a reading-list-commands file, then we also see a CloudKit folder with an sqlite database named Records.db. Lets look at these files in editors, beginning with “reading-list”. 

Looks like a binary file made with NSKeyedArchiver and contains the list of articleIDs and date added. The file named “reading-list-commands” just looks like an array of article IDs stored in binary, my guess would be user actions that need sent to the server are cached in here. At first glance flat files looks really, really bad, it means all the data is stored in memory and written out whenever it changes. Its possible they chose flat files rather than a database since fetch changes might error CKServerTokenExpired, which documentation says to toss the cache and start with a fresh download using a nil token, so in that case it does make sense to just delete a file rather than empty and rebuild a database, it would be good to know how common this scenario is tho, Apple definitely to need provide more information to what at moment is very black-box like. To aid my development, I have asked on Stack Overflow if there is a way to simulate CKServerTokenExpired. The best alternative to flat files for the model is CoreData, with automatic UI updates and table sorting, so they would need a very good reason for us to give all that up and use flat files, or maybe they just didn’t have time?

So now we know they are using flat files for all the private syncing which is very interesting given other developers have attempted to sync to Core Data. Lets see what secrets are hiding in the Records.db by opening it in a SQLite editor.

No Core Data here either! They way to tell is lots of capitalised “Z_” prefixed tables and fields, so here they are using sqlite directly. Also we notice the recordID is being encoded with a colon seperator, e.g. recordName:zone:owner, which is interesting because I’ve seen other developers attempt to encode all the properties of the recordID in different ways, some even storing different zones in different databases. The owner might be the creatorUserRecordID.recordName (or maybe modifier) because usually it is __defaultOwner__ when looking at that record name of records your own account creates, rather than being your own user UUID. This table even contains the containerIdentifier which you would think would be redundant information, since the app knows what containers it contacts so this reminded me the last time I looked at the CloudKit headers I did see Sqlite mentioned, lets open that now.

In the file list on the left we see a CKSQLite class which is an Obj-C wrapper around the sqlite library. Opening CKRecordID we see it has methods sqliteRepresentation and initWithSqliteRepresentation likely for the colon separator parsing. Finally, by searching for what class is using CKSQLite we find a large class CKPackage (pictured), which looks like it is responsible for the database we saw and caching all the records. How it is actually used isn’t clear, like if it is loosely-coupled to the operations, in that they decide which records get cached, or if it is tightly-coupled in that the operations automatically are caching records. That would require more investigation but it could suggest caching features are coming to CloudKit APIs of the future. But at least we know now they are using flat files for sync of uploads and downloads of private records, and sqlite for caching public records. This is sad in a way that we didn’t see any core data database perhaps with a sync status flag we could have looked to for inspiration, but also reminds us not to over think a problem and using tried-and-tested techniques of saving files is still fine.

Conclusion

There we have it, it’s been a long journey but we have learned a lot. Now we know the two classes we should be focussing on to peform a sync are the CKFetchDatabaseChangesOperation and CKFetchRecordZoneChangesOperation. Also we know for realtime sync we should be using silent push notifications via CKRecordZoneSubscription and CKNotificationInfo with shouldSendContentAvailable, and upon push received we should first sync database changes and then only the record zone changes for the zones included. We also know we should implement a manual refresh just in case push notifications don’t arrive. We learned about using flat files for caching private database sync, and how CKRecords could be encoded in sqlite database fields. Did we learn anything else? Probably! In the future, we might see if they coalesce sync requests originating from multiple pushes. Another question that came to mind is if if they re-download records to the same device that created it, which is a common dilemma in sync solutions. I hope this post has helped and now you are starting off on the right foot for building the perfect multi-device sync app!

Footnote

There is one final thing I’d like to mention concerning a new feature of the iOS 10 API. They added the ability for the sync classes to repeat themselves to get all data, via the fetchAllChanges properties. This is great news since the big developer complaints with the CloudKit API is it was very complicated to make the required repeat requests. The strange thing is they only added it to the classes involved in syncing, not to CKQueryOperation for example. On the one hand this shows Apple’s focus with CloudKit is towards improving syncing, which is great, but it also is bad in that they got the API wrong the first time, and have subsequently had to rename classes ( CKFetchRecordChangesOperation -> CKFetchRecordZoneChangesOperation) and add essential properties like fetchAllChanges. It also seems rushed compared to normal, like a block on CKFetchRecordZoneChangesOperation is named recordZoneChangeTokensUpdatedBlock that pluralisation just seems strange to me, i.e. a zone only has one token, and stands out as inconsistent with the other names used. This is the kind of thing that should get cleaned up as they iterate over the API design, maybe Scott Forstall’s perfectionist strategy of iterating API designs ten times before release is no longer being implemented.

ObjC Delegation and Notifications

I was researching the different ways people are implementing the part of MVC where the model notifies the controller about new data. A limitation of the delegate pattern is only one class can be the delegate, for other classes to listen then they need to use Notification Center notifications (or use a custom multi-cast delegate). Normally when the model class is called to say, insert an object, it would use the standard routine to check the delegate responds to the selector, and then call the delegate method as defined on the protocol. Then when reading some Apple’s delegations article I noticed something rather interesting:

Delegation and Notifications
The delegate of most Cocoa framework classes is automatically registered as an observer of notifications posted by the delegating object. The delegate need only implement a notification method declared by the framework class to receive a particular notification message. Following the example above, a window object posts an NSWindowWillCloseNotification to observers but sends a windowShouldClose: message to its delegate.

The interesting point here is that the delegate is registered as an observer, which is different from calling the delegate method directly. To check this was a correct statement I went to the gold mine that is the GNUStep source code for NSWindow (Github Mirror) and found this very nifty piece of code:

- (void) setDelegate: (id)anObject
{
  if (_delegate)
    {
      [nc removeObserver: _delegate name: nil object: self];
    }
  _delegate = anObject;
#define SET_DELEGATE_NOTIFICATION(notif_name) \
if ([_delegate respondsToSelector: @selector(window##notif_name:)]) \
[nc addObserver: _delegate \
selector: @selector(window##notif_name:) \
name: NSWindow##notif_name##Notification object: self]

SET_DELEGATE_NOTIFICATION(DidBecomeKey);
SET_DELEGATE_NOTIFICATION(DidBecomeMain);
SET_DELEGATE_NOTIFICATION(DidChangeScreen);
SET_DELEGATE_NOTIFICATION(DidDeminiaturize);
SET_DELEGATE_NOTIFICATION(DidExpose);
SET_DELEGATE_NOTIFICATION(DidMiniaturize);
SET_DELEGATE_NOTIFICATION(DidMove);
SET_DELEGATE_NOTIFICATION(DidResignKey);
SET_DELEGATE_NOTIFICATION(DidResignMain);
SET_DELEGATE_NOTIFICATION(DidResize);
SET_DELEGATE_NOTIFICATION(DidUpdate);
SET_DELEGATE_NOTIFICATION(WillClose);
SET_DELEGATE_NOTIFICATION(WillMiniaturize);
SET_DELEGATE_NOTIFICATION(WillMove);
}

So sure enough, it adds the delegate as an observer, and only doing so if it responds to the selector. It’s is a pretty good optimisation because now at each event only the notification needs to be posted. Previously, the delegate performing the selector would be checked every time (still done this way for delegate methods that contain data). I also really like the macro. The only problem is the notifications are rather primitive in that they contain no data about the reason, they only contain the object self. In seeing this implementation it sparked my interest to investigate if Apple implemented it the same way, which could be tested by removing the delegate as an observer and then seeing if the delegate method is no longer called, don’t think I’ll bother just now though.

NSArray @property backed by a NSMutableArray

Saw a bunch of posts today on Stack Overflow asking how to implement this:

NSArray @property backed by a NSMutableArray

Hiding privately mutable properties behind immutable interfaces in Objective-C (my answer)

Protect from adding object to NSMutableArray in public interface

Having readonly nsarray property and nsmutable array not readonly with the same name and _ in xcode 4.2 vs 4.5

How to expose an NSMutableArray as an NSArray as a return type from a method

The main reason is to expose an array as immutable in the public interface, to convey that it shouldn’t be modified externally, but can be mutated internally. This is a typical thing you would want to do in a Model class, e.g. having an insertObject method and then notifying controllers via a delegate and notifications. One of the stumbling blocks I noticed is people were trying to implement is solely with their knowledge of properties, where you can redefine a readonly property as readwrite internally using a category, unfortunately you cannot change its type. With some understanding of ObjC before automatic synthesis the solution is to manually implement the ivar that backs the property, like this:

.h file:

@interface Model : NSObject

@property (nonatomic, strong) NSArray *objects;

-(void)insertObject:(id)object

// todo: delegate
@end

.m file:

@implementation Model{
    NSMutableArray* _objects;
}

-(void)insertObject:(id)object{
    if(!_objects){
        _objects = [[NSMutableArray alloc] init];
    }
    [_objects addObject:object];
}

@end

Just one of the many reasons developers have such a difficulty extracting their model from the view controller in Apple’s default project templates.

NSOperationQueue maxConcurrentOperationCount 1 Serial Queue Problem

Around the web and even in Apple’s WWDC 2015 Advanced NSOperations talk they say:

“If we set the max concurrent operation count of an NSOperationQueue to be 1, then we essentially make our NSOperationQueue into a serial operation queue.”

Sometimes developers also refer to this configuration as a first in first out (FIFO) queue. It works ok in very simple cases but by ignoring operation dependencies things can go bad fast. To illustrate what I mean, see this sample code:

    NSOperationQueue *queue = [[NSOperationQueue alloc] init];
    queue.maxConcurrentOperationCount = 1;
    
    NSBlockOperation *op1 = [NSBlockOperation blockOperationWithBlock:^{
        NSLog(@"op1");
    }];
    
    NSBlockOperation *op2 = [NSBlockOperation blockOperationWithBlock:^{
        NSLog(@"op2");
    }];
    
    NSBlockOperation *op3 = [NSBlockOperation blockOperationWithBlock:^{
        NSLog(@"op3");
    }];
    
    NSBlockOperation *waitOp = [NSBlockOperation blockOperationWithBlock:^{
        NSLog(@"op4");
    }];
    
    // uncomment to fix problem
    // [op2 addDependency:op1];
    // [op3 addDependency:op2];

    [op2 addDependency: waitOp];
    
    [queue addOperations:@[op1, op2, op3] waitUntilFinished:NO];

The output from this code is:

2017-01-07 22:11:43.322770 OpTest[24703:35783606] op1
2017-01-07 22:11:43.322846 OpTest[24703:35783564] op3

What has happened is since op2 has a dependency on waitOp (which has been made unable to be started), op3 has skipped the queue and jumped ahead of op2. If it was a serial queue then you might expect only “op1” to be output and the queue waiting on waitOp to finish to then start op2.

By uncommenting the 2 addDependency lines we then get the expected behaviour:

2017-01-07 22:27:23.876388 OpTest[24909:35842006] op1

The funny thing is that with the dependencies set up correctly we can now remove the line that sets the maxConcurrentOperationCount and the behaviour stays correct. The problem is that usually developers want to retry their operations which does require understanding dependencies in which case I believe it would have been better to learn them from the outset rather than use this trick. Also if they do then they will be one step closer to learning the real power which is cross-queue dependencies.

YapDatabase Disappointing CloudKit Example

It’s not often you see code as disappointing as this, particularly because CloudKit is so well designed for the situation being described, but it’s actually very common. I think it happens when you have a developer with speciality in one area, trying to apply it to their understanding of how to implement something in a new area, and just getting the job done as quickly as possible.

I was searching Github for examples of how people are handling CKErrorChangeTokenExpired, and came across this CloudKitManager.m by YapDatabase. In scrolling through I noticed this function that looks to be chaining some CloudKit setup operations together:

- (void)continueCloudKitFlow
{
    DDLogInfo(@"%@ - %@", THIS_FILE, THIS_METHOD);
    
    if (self.needsCreateZone)
    {
        [self createZone];
    }
    else if (self.needsCreateZoneSubscription)
    {
        [self createZoneSubscription];
    }
    else if (self.needsFetchRecordChangesAfterAppLaunch)
    {
        [self fetchRecordChangesAfterAppLaunch];
    }

First thought is, since these methods are usually all async and dependent on one another how can they all be called in line like this? Taking a look at one of these methods gives an ugly surprise! (And I don’t mean them using tabs instead of spaces but yeh that’s ugly too! Had to even fix them just to paste into WordPress.)

- (void)createZone
{
    dispatch_async(setupQueue, ^{ @autoreleasepool {
        
        // Suspend the queue.
        // We will resume it upon completion of the operation.
        // This ensures that there is only one outstanding operation at a time.
        dispatch_suspend(setupQueue);
        
        [self _createZone];
    }});
}

So all of the methods are called one after another instantly, but to make the actual methods (prefixed with underscore) run serially, the developer runs them as blocks on a GCD dispatch queue, and when each block starts it pauses the  queue to prevent the other ones being called. Unbelievable!

The question you are all asking is why on earth didn’t the developer simply make use of CloudKit’s beautiful use of NSOperations and just set up dependencies via an NSOperationQueue to run them serially automatically? I would guess the answer is this particular developer is an expert in GCD (based on the information in the Readme of the project CocoaAsyncSockets), so put simply, they used what they knew to solve the problem.

It’s one of those situations where they got the job done, so that’s great, but I think if everyone takes a moment to learn the NSOperation API properly and see how amazingly well CloudKit builds upon it, it would bring much more enjoyment to developers.

In case you were wondering how it should be done, is the CloudKit operations don’t have to be added to a database or container via addOperation to be processed, a database property can be set on the operations and then it can be added to a custom operation queue, thus allows you to group your operations together, and with dependencies order them serially. It is easy to miss but in the CKDatabase header it says “or schedule operations on your own queue”, that’s the secret.

Xcode 8 iOS 10 Non-Stop Device Logs

This post is about my Stack Overflow answer.

In Xcode, Window->Devices, choosing a device and clicking the up arrow in the bottom bar brings up the console and this traditionally allowed you to view output from your app when not running in the debugger, e.g. for testing background launching behaviour. Unfortunately as of Xcode 8 and iOS 10 there is now a lot more log output – very low level debug logging is on so hundreds of lines pass by every second, making it impossible to see the logs you are interested in.

The solution is to upgrade to macOS Sierra, the new Console app now displays connected devices, and it allows filtering the log by many different categories, process is probably the most handy where you can simply enter the name of your app’s executable. This now makes the Xcode Devices log redundant, it never was very good at search anyway.