Investigating CloudKit Sync in the Apple News App

Introduction

Syncing data across a user’s own devices in a privacy-sensitive way is one of the flagship features of CloudKit. Unfortunately many agree “it isn’t as documented as it might be”, e.g. there is no sync sample from Apple, the WWDC 2014 talk and WWDC 2015 talk didn’t quite get to the topic although did offer some vague hints, in WWDC 2016 talk the presenters gave lots of unfinished sentences possibly due to lack of time, e.g. <sic>you have to do it this way “for a variety of reasons.” Looking at the CloudKit framework headers none of the classes have sync in their name, or even have the word “sync” mentioned. Slightly better is that more recently the class doc for CKFetchRecordChangesOperation has been updated to mention this is the one to use for synchronising, however having information spread across multiple places is really hard work. Furthermore, fully understanding CloudKit behaviour requires knowledge of other things like NSOperation quality of service and NSURLSession discretionary. To add to the confusion, the open-source projects out there are all attempting to achieve sync in different ways, e.g. some are attempting to sync with the public database which is missing the required features, or they might use a periodic sync rather than realtime which CloudKit was designed for using push notifications, and zones are a really stumbling block. These common mistakes is clearly a result of lack of good samples and documentation, and in particular lack of some absolutely vital information, for example the CKFetchRecordZoneChangesOperation. I happened come across a Stack Overflow answer by a CloudKit engineer who shared that not all the changes are returned, they are coalesced to remove unnecessary ones, e.g. if a record was added and then deleted since the previous request (tracked by a token) it isn’t included. This was quite eye-opening because it shows the server is a lot smarter than expected, and knowing this now yes this could be used for an efficient sync. What also helped me is, as we see later, they do hit a HTTP endpoint that is named sync, which helped confirm this must be the right path. I think Apple could have put the words sync in the class name or at the very least put in the header, use this class for syncing!

In a situation like this where there is much ambiguity it is useful to look at how Apple do things, for some ground truth, and they are using CloudKit sync in the News App so lets take an in-depth look at that and using two devices for testing, an iPod Touch and iPhone 6s both on 10.1.1.

News Article Download

When the app starts up it shows news articles, since all users can see these articles we would expect them to be in the public database. We will use a web proxy to analyse the requests, this won’t give us exact detail of how the framework classes are being configured, but it will give us an idea of the general algorithm. We will be using the 6s to monitor the requests.

As we can see it performs a query to the container com.apple.news.public which is the public database. The record contains and articleID, title, thumbnail, contentURL etc. so the full article is downloaded in separate requests that go directly to the news organisations own servers. The thumbnail is a URL to an image on the icloud-content server you can see in the screenshot. The query used contains minOrder and maxOrder which are integers so would suggest they are doing a query to get new articles, this isn’t a sync its usually called a delta download, i.e. only downloads the new information. The articles have an order field like an auto-inc, or sequence number, newer articles have a higher number, which is better than using timestamps where 2 articles might have the same time. This is possible because only Apple is the one inserting records. This kind of delta download is a great bandwidth saver however it only allows only new articles to be downloaded which has a limitation you can see in the below screenshots.

 

As you can see the first article had its title changed between 4h ago and 13h ago. The limitation of their design is it doesn’t allow old articles to be updated, which doesn’t fit well with the news industry where headlines can be changed frequently as a result of errors or updated information. This would require existing downloaded articles to be updated, perhaps even deleted. This isn’t possible with the public database because it doesn’t have the required features to support a full sync. The alternative design would be to clear the cache and re-download all the articles every time, which would ensure the user is seeing the latest list, however that may have higher bandwidth requirements. Apple must have run the numbers for their number of records and data involved and decided an append only delta download was the way to go. There is another interesting usability feature here, as you can see we are on the history list, if a user is browsing the list to find a previously read article it certainly would make it harder to find if the title was changed. Now it becomes a very interesting problem, because you have a trade off between what is technically optimal with what is best for users.

Bookmark & Reading List Sync

Now that we have covered how articles are downloaded lets now look at the features we are really interested in, how it performs the sync between devices. The News app has two features that are synced, bookmarks and history. The history view has already been shown in the above screenshots. An item is added to the history after an article is viewed and the user has scrolled down a bit, or maybe spends some time in the article, or perhaps a combination of both. Articles can be bookmarked when viewing them, by tapping the bookmark icon on the bottom right. Now the really interesting part, if you have two devices side-by-side these two sets of data are updated almost instantly (~5 seconds) when changes happen. For example bookmark an article on one device and it appears in the bookmark list on the other device, un-bookmark the article and it disappears from the other devices list. So there we have the feature we are looking for, a full sync between devices so lets see how it is implemented. We will tackle the push part later, we’ll focus on the News App’s requests just now though.

So we have the 6s connected to the proxy and open at the saved articles page. On the iPod touch we open an article and bookmark it. 5 seconds later this happens:

As we can see in the first request it hits an end point called “zone/sync” to a container called com.apple.news.private. So now we know they are using both public and private databases for this app, which is interesting to me because obviously articles need to be referenced, and we were told many times CKReference doesn’t work across zones or databases, all they’ve done here is simply make the articleID a string field rather than a reference, I suppose they aren’t bothered about referential integrity. Next fortunately for us a familiar looking class name is included in the request, looks like a CKFetchDatabaseChangesOperation (the one in the log has prefix CKD which is because the actual request goes through the CloudKit daemon, so its like an RPC or remote class). On the response tab (not shown in screenshot) we see ReadingList and ReadingHistory which are the name of the two zones that have changed. Lets take a look at what happens next:

These next two requests are to the endpoint “record/sync” again to the private container and again we can see the class looks like the familiar CKFetchRecordZoneChangesOperation (Note. pre iOS 10 the class was CKFetchRecordChangesOperation). This first sync request contains the zone name ReadingList and as expected the second contains ReadingHistory. From more testing we see that the record/sync is only requested if the zone name is contained in the zone/sync.

Background Sync

Another feature is these lists are already up-to-date when the app is re-opened. If the app is killed and restarted then it already has the previous data which sparks my interest to see how they are achieving caching, but for now we will focus on what technique they are using for updating e.g. update on coming to foreground, background fetch or push notification. So to find out we can test this with the 6s connected to the proxy just on the homescreen, and then on the iPod Touch using news to read and bookmark an article. We’ll use the proxy and also the new Sierra Console so we can gain an insight into what the 6s is doing.

A push notification! And it has the content-available flag this shows they are using silent push, implemented using a CKNotificationInfo with shouldSendContentAvailable set. We also also see the private container and the zid which is the short version of Zone Name. Apple shorten the json key names in pushes because packet size is limited. Because the push contains the zoneID this would suggest they are using CKRecordZoneSubscription. Now lets see what the proxy logged:

Looks like the exact same requests as when the app is in the foreground. Finding out what zones changed and then what records within them changed. In fact the same push is used when the app is in the foreground, so now we know how the info is kept up-to-date. I don’t know about you but what crosses my mind is if multiple pushes are received do they coalesce the fetch changes requests? I might investigate that later on.

Pull-to-Refresh

Next, I noticed there is also a pull-to-refresh enabled on the bookmark and history table views. This was perhaps implemented as a fall-back in case for some reason the push notification doesn’t arrive so it allows the user to force a refresh of new data. Or maybe they are using the feature to clear the cache and re-download all the bookmarks to clear up any inconsistencies? Lets do a pull-to-refresh on the 6s and see what the proxy shows:

We see only a CKFetchRecordZoneChanges this time, no record download. Scrolling down the request shows it is for the ReadingList zone name. Similarly if we pull-to-refresh on the History tab we see the same request but for the ReadingHistory zone name. This proves they are only using this feature to do a sync, so as a replacement for a missing push notification, rather than a full clear-cache and re-download everything.

Caching

Now it wouldn’t be a proper sync without caching, this allows the app to be killed and restarted and still show the previous info. So lets see how they achieved that. To that we will use a jailbroken iPod Touch on iOS 9.3.3 so there is a chance that the caching has changed on iOS 10 but hopefully this is still interesting. We will connect over SSH and browse the file system to find where the News app stores its data.

The data is in /private/var/mobile/Container/Data/Application/ContainerID which was found by a process of elimination where first the mobile user’s Library folder was browsed and when nothing was found there I looked to the containers. I think this is a relatively new concept of storing platform (or built-in) app data in the container folder. Since container folders have a UUID it can be tricky to find the right one, how I achieve it is modify something in the app, like save an article to the reading list, and then sort all the containers by date. So in the screenshot above we have found the folder and we can see the private data in a folder, where i have highlighted the reading-list file, there is also a reading-list-commands file, then we also see a CloudKit folder with an sqlite database named Records.db. Lets look at these files in editors, beginning with “reading-list”. 

Looks like a binary file made with NSKeyedArchiver and contains the list of articleIDs and date added. The file named “reading-list-commands” just looks like an array of article IDs stored in binary, my guess would be user actions that need sent to the server are cached in here. At first glance flat files looks really, really bad, it means all the data is stored in memory and written out whenever it changes. Its possible they chose flat files rather than a database since fetch changes might error CKServerTokenExpired, which documentation says to toss the cache and start with a fresh download using a nil token, so in that case it does make sense to just delete a file rather than empty and rebuild a database, it would be good to know how common this scenario is tho, Apple definitely to need provide more information to what at moment is very black-box like. To aid my development, I have asked on Stack Overflow if there is a way to simulate CKServerTokenExpired. The best alternative to flat files for the model is CoreData, with automatic UI updates and table sorting, so they would need a very good reason for us to give all that up and use flat files, or maybe they just didn’t have time?

So now we know they are using flat files for all the private syncing which is very interesting given other developers have attempted to sync to Core Data. Lets see what secrets are hiding in the Records.db by opening it in a SQLite editor.

No Core Data here either! They way to tell is lots of capitalised “Z_” prefixed tables and fields, so here they are using sqlite directly. Also we notice the recordID is being encoded with a colon seperator, e.g. recordName:zone:owner, which is interesting because I’ve seen other developers attempt to encode all the properties of the recordID in different ways, some even storing different zones in different databases. The owner might be the creatorUserRecordID.recordName (or maybe modifier) because usually it is __defaultOwner__ when looking at that record name of records your own account creates, rather than being your own user UUID. This table even contains the containerIdentifier which you would think would be redundant information, since the app knows what containers it contacts so this reminded me the last time I looked at the CloudKit headers I did see Sqlite mentioned, lets open that now.

In the file list on the left we see a CKSQLite class which is an Obj-C wrapper around the sqlite library. Opening CKRecordID we see it has methods sqliteRepresentation and initWithSqliteRepresentation likely for the colon separator parsing. Finally, by searching for what class is using CKSQLite we find a large class CKPackage (pictured), which looks like it is responsible for the database we saw and caching all the records. How it is actually used isn’t clear, like if it is loosely-coupled to the operations, in that they decide which records get cached, or if it is tightly-coupled in that the operations automatically are caching records. That would require more investigation but it could suggest caching features are coming to CloudKit APIs of the future. But at least we know now they are using flat files for sync of uploads and downloads of private records, and sqlite for caching public records. This is sad in a way that we didn’t see any core data database perhaps with a sync status flag we could have looked to for inspiration, but also reminds us not to over think a problem and using tried-and-tested techniques of saving files is still fine.

Conclusion

There we have it, it’s been a long journey but we have learned a lot. Now we know the two classes we should be focussing on to peform a sync are the CKFetchDatabaseChangesOperation and CKFetchRecordZoneChangesOperation. Also we know for realtime sync we should be using silent push notifications via CKRecordZoneSubscription and CKNotificationInfo with shouldSendContentAvailable, and upon push received we should first sync database changes and then only the record zone changes for the zones included. We also know we should implement a manual refresh just in case push notifications don’t arrive. We learned about using flat files for caching private database sync, and how CKRecords could be encoded in sqlite database fields. Did we learn anything else? Probably! In the future, we might see if they coalesce sync requests originating from multiple pushes. Another question that came to mind is if if they re-download records to the same device that created it, which is a common dilemma in sync solutions. I hope this post has helped and now you are starting off on the right foot for building the perfect multi-device sync app!

Footnote

There is one final thing I’d like to mention concerning a new feature of the iOS 10 API. They added the ability for the sync classes to repeat themselves to get all data, via the fetchAllChanges properties. This is great news since the big developer complaints with the CloudKit API is it was very complicated to make the required repeat requests. The strange thing is they only added it to the classes involved in syncing, not to CKQueryOperation for example. On the one hand this shows Apple’s focus with CloudKit is towards improving syncing, which is great, but it also is bad in that they got the API wrong the first time, and have subsequently had to rename classes ( CKFetchRecordChangesOperation -> CKFetchRecordZoneChangesOperation) and add essential properties like fetchAllChanges. It also seems rushed compared to normal, like a block on CKFetchRecordZoneChangesOperation is named recordZoneChangeTokensUpdatedBlock that pluralisation just seems strange to me, i.e. a zone only has one token, and stands out as inconsistent with the other names used. This is the kind of thing that should get cleaned up as they iterate over the API design, maybe Scott Forstall’s perfectionist strategy of iterating API designs ten times before release is no longer being implemented.

ObjC Delegation and Notifications

I was researching the different ways people are implementing the part of MVC where the model notifies the controller about new data. A limitation of the delegate pattern is only one class can be the delegate, for other classes to listen then they need to use Notification Center notifications (or use a custom multi-cast delegate). Normally when the model class is called to say, insert an object, it would use the standard routine to check the delegate responds to the selector, and then call the delegate method as defined on the protocol. Then when reading some Apple’s delegations article I noticed something rather interesting:

Delegation and Notifications
The delegate of most Cocoa framework classes is automatically registered as an observer of notifications posted by the delegating object. The delegate need only implement a notification method declared by the framework class to receive a particular notification message. Following the example above, a window object posts an NSWindowWillCloseNotification to observers but sends a windowShouldClose: message to its delegate.

The interesting point here is that the delegate is registered as an observer, which is different from calling the delegate method directly. To check this was a correct statement I went to the gold mine that is the GNUStep source code for NSWindow (Github Mirror) and found this very nifty piece of code:

- (void) setDelegate: (id)anObject
{
  if (_delegate)
    {
      [nc removeObserver: _delegate name: nil object: self];
    }
  _delegate = anObject;
#define SET_DELEGATE_NOTIFICATION(notif_name) \
if ([_delegate respondsToSelector: @selector(window##notif_name:)]) \
[nc addObserver: _delegate \
selector: @selector(window##notif_name:) \
name: NSWindow##notif_name##Notification object: self]

SET_DELEGATE_NOTIFICATION(DidBecomeKey);
SET_DELEGATE_NOTIFICATION(DidBecomeMain);
SET_DELEGATE_NOTIFICATION(DidChangeScreen);
SET_DELEGATE_NOTIFICATION(DidDeminiaturize);
SET_DELEGATE_NOTIFICATION(DidExpose);
SET_DELEGATE_NOTIFICATION(DidMiniaturize);
SET_DELEGATE_NOTIFICATION(DidMove);
SET_DELEGATE_NOTIFICATION(DidResignKey);
SET_DELEGATE_NOTIFICATION(DidResignMain);
SET_DELEGATE_NOTIFICATION(DidResize);
SET_DELEGATE_NOTIFICATION(DidUpdate);
SET_DELEGATE_NOTIFICATION(WillClose);
SET_DELEGATE_NOTIFICATION(WillMiniaturize);
SET_DELEGATE_NOTIFICATION(WillMove);
}

So sure enough, it adds the delegate as an observer, and only doing so if it responds to the selector. It’s is a pretty good optimisation because now at each event only the notification needs to be posted. Previously, the delegate performing the selector would be checked every time (still done this way for delegate methods that contain data). I also really like the macro. The only problem is the notifications are rather primitive in that they contain no data about the reason, they only contain the object self. In seeing this implementation it sparked my interest to investigate if Apple implemented it the same way, which could be tested by removing the delegate as an observer and then seeing if the delegate method is no longer called, don’t think I’ll bother just now though.

NSArray @property backed by a NSMutableArray

Saw a bunch of posts today on Stack Overflow asking how to implement this:

NSArray @property backed by a NSMutableArray

Hiding privately mutable properties behind immutable interfaces in Objective-C (my answer)

Protect from adding object to NSMutableArray in public interface

Having readonly nsarray property and nsmutable array not readonly with the same name and _ in xcode 4.2 vs 4.5

How to expose an NSMutableArray as an NSArray as a return type from a method

The main reason is to expose an array as immutable in the public interface, to convey that it shouldn’t be modified externally, but can be mutated internally. This is a typical thing you would want to do in a Model class, e.g. having an insertObject method and then notifying controllers via a delegate and notifications. One of the stumbling blocks I noticed is people were trying to implement is solely with their knowledge of properties, where you can redefine a readonly property as readwrite internally using a category, unfortunately you cannot change its type. With some understanding of ObjC before automatic synthesis the solution is to manually implement the ivar that backs the property, like this:

.h file:

@interface Model : NSObject

@property (nonatomic, strong) NSArray *objects;

-(void)insertObject:(id)object

// todo: delegate
@end

.m file:

@implementation Model{
    NSMutableArray* _objects;
}

-(void)insertObject:(id)object{
    if(!_objects){
        _objects = [[NSMutableArray alloc] init];
    }
    [_objects addObject:object];
}

@end

Just one of the many reasons developers have such a difficulty extracting their model from the view controller in Apple’s default project templates.

NSOperationQueue maxConcurrentOperationCount 1 Serial Queue Problem

Around the web and even in Apple’s WWDC 2015 Advanced NSOperations talk they say:

“If we set the max concurrent operation count of an NSOperationQueue to be 1, then we essentially make our NSOperationQueue into a serial operation queue.”

Sometimes developers also refer to this configuration as a first in first out (FIFO) queue. It works ok in very simple cases but by ignoring operation dependencies things can go bad fast. To illustrate what I mean, see this sample code:

    NSOperationQueue *queue = [[NSOperationQueue alloc] init];
    queue.maxConcurrentOperationCount = 1;
    
    NSBlockOperation *op1 = [NSBlockOperation blockOperationWithBlock:^{
        NSLog(@"op1");
    }];
    
    NSBlockOperation *op2 = [NSBlockOperation blockOperationWithBlock:^{
        NSLog(@"op2");
    }];
    
    NSBlockOperation *op3 = [NSBlockOperation blockOperationWithBlock:^{
        NSLog(@"op3");
    }];
    
    NSBlockOperation *waitOp = [NSBlockOperation blockOperationWithBlock:^{
        NSLog(@"op4");
    }];
    
    // uncomment to fix problem
    // [op2 addDependency:op1];
    // [op3 addDependency:op2];

    [op2 addDependency: waitOp];
    
    [queue addOperations:@[op1, op2, op3] waitUntilFinished:NO];

The output from this code is:

2017-01-07 22:11:43.322770 OpTest[24703:35783606] op1
2017-01-07 22:11:43.322846 OpTest[24703:35783564] op3

What has happened is since op2 has a dependency on waitOp (which has been made unable to be started), op3 has skipped the queue and jumped ahead of op2. If it was a serial queue then you might expect only “op1” to be output and the queue waiting on waitOp to finish to then start op2.

By uncommenting the 2 addDependency lines we then get the expected behaviour:

2017-01-07 22:27:23.876388 OpTest[24909:35842006] op1

The funny thing is that with the dependencies set up correctly we can now remove the line that sets the maxConcurrentOperationCount and the behaviour stays correct. The problem is that usually developers want to retry their operations which does require understanding dependencies in which case I believe it would have been better to learn them from the outset rather than use this trick. Also if they do then they will be one step closer to learning the real power which is cross-queue dependencies.

YapDatabase Disappointing CloudKit Example

It’s not often you see code as disappointing as this, particularly because CloudKit is so well designed for the situation being described, but it’s actually very common. I think it happens when you have a developer with speciality in one area, trying to apply it to their understanding of how to implement something in a new area, and just getting the job done as quickly as possible.

I was searching Github for examples of how people are handling CKErrorChangeTokenExpired, and came across this CloudKitManager.m by YapDatabase. In scrolling through I noticed this function that looks to be chaining some CloudKit setup operations together:

- (void)continueCloudKitFlow
{
    DDLogInfo(@"%@ - %@", THIS_FILE, THIS_METHOD);
    
    if (self.needsCreateZone)
    {
        [self createZone];
    }
    else if (self.needsCreateZoneSubscription)
    {
        [self createZoneSubscription];
    }
    else if (self.needsFetchRecordChangesAfterAppLaunch)
    {
        [self fetchRecordChangesAfterAppLaunch];
    }

First thought is, since these methods are usually all async and dependent on one another how can they all be called in line like this? Taking a look at one of these methods gives an ugly surprise! (And I don’t mean them using tabs instead of spaces but yeh that’s ugly too! Had to even fix them just to paste into WordPress.)

- (void)createZone
{
    dispatch_async(setupQueue, ^{ @autoreleasepool {
        
        // Suspend the queue.
        // We will resume it upon completion of the operation.
        // This ensures that there is only one outstanding operation at a time.
        dispatch_suspend(setupQueue);
        
        [self _createZone];
    }});
}

So all of the methods are called one after another instantly, but to make the actual methods (prefixed with underscore) run serially, the developer runs them as blocks on a GCD dispatch queue, and when each block starts it pauses the  queue to prevent the other ones being called. Unbelievable!

The question you are all asking is why on earth didn’t the developer simply make use of CloudKit’s beautiful use of NSOperations and just set up dependencies via an NSOperationQueue to run them serially automatically? I would guess the answer is this particular developer is an expert in GCD (based on the information in the Readme of the project CocoaAsyncSockets), so put simply, they used what they knew to solve the problem.

It’s one of those situations where they got the job done, so that’s great, but I think if everyone takes a moment to learn the NSOperation API properly and see how amazingly well CloudKit builds upon it, it would bring much more enjoyment to developers.

In case you were wondering how it should be done, is the CloudKit operations don’t have to be added to a database or container via addOperation to be processed, a database property can be set on the operations and then it can be added to a custom operation queue, thus allows you to group your operations together, and with dependencies order them serially. It is easy to miss but in the CKDatabase header it says “or schedule operations on your own queue”, that’s the secret.

Xcode 8 iOS 10 Non-Stop Device Logs

This post is about my Stack Overflow answer.

In Xcode, Window->Devices, choosing a device and clicking the up arrow in the bottom bar brings up the console and this traditionally allowed you to view output from your app when not running in the debugger, e.g. for testing background launching behaviour. Unfortunately as of Xcode 8 and iOS 10 there is now a lot more log output – very low level debug logging is on so hundreds of lines pass by every second, making it impossible to see the logs you are interested in.

The solution is to upgrade to macOS Sierra, the new Console app now displays connected devices, and it allows filtering the log by many different categories, process is probably the most handy where you can simply enter the name of your app’s executable. This now makes the Xcode Devices log redundant, it never was very good at search anyway.

CKRecordZoneSubscription Notes

If you attempt to set desiredKeys on a notificationInfo for the new iOS 10 CKRecordZoneSubscription class, saving results in the runtime error:

<CKError 0x17424b610: "Invalid Arguments" (12/2006); server message = "cannot add additionalFields to this subscription type"; uuid = 4E08C616-97AA-4E9A-B584-B7972B1CD99B; container ID = "iCloud.com.x">

Which makes sense given the notification is about a zone changing and not a record, however one might assume it can send the record that was responsible for the change but looking to CKRecordZoneNotification shows no record properties are available. It’s always kind of disappointing to see subclassing not working out, i.e. where the subclass is denied features of the parent, it can sometimes point to bad class design however in this case Apple could have handled it better either by client side validation or at least a note in the header. It’s extra disappointing given in iOS 10 they refactored from flat CKSubscription inits for the different subscription types (which did have their own limitations, e.g. initWithZoneID only allowed zero for options) to subclasses. I hope they’ve thought this new design through because if it is refactored again it could be a real headache for framework backwards compatibility.

 

CloudKit Syncing References

Here are some references from around the web on how to implement CloudKit syncing. It will be added to…

9th August 2017

In the Build Better Apps with CloudKit Dashboard WWDC 2017 video at 4:30 Dave Browning presents a ToDo list app that syncs from Core Data with CloudKit 6:15. I wrote to Dave in June and he told me that they do plan to release the source for that app, they just need to clean it up a bit and review it internally first. He said he’d let me know once it has been published but it has been a few months now. I’ll update this post if it gets released.

5th January 2017

As I Learn CloudKit Syncing by Eric Allam

Probably my favourite tutorial so far, Part 1 follows the example given in the Advanced CloudKit talk from 2014, unfortunately the talk and this tutorial doesn’t get as far as using push notifications via a CKRecordZoneSubscription and resorts to fetch only. It correctly uses CKFetchRecordChangesOperation for this, however it contains a mistake in part 4. It says that only changed properties are included in the CKRecord given by the recordChangedBlock, in fact the full record is included by default, and it now also supports a desiredKeys in the newer version of this class named CKFetchRecordZoneChangesOperation. It correctly identifies using NSOperation dependencies as the right approach for queuing related operations, however in seek of queue error handling Eric looks to the Advanced NSOperations talk and corresponding open source project, which unfortunately is of a different design to CloudKit NSOperations and he hits some problems.

5th January 2017

Seam – Seamless CloudKit Sync with CoreData

This project provides tight coupling with CoreData that seems to go against Apple’s designs of keeping frameworks more loose. For example, it wraps conflict handling which might be something the developer requires full control over. It’s always the same with wrapping, you end up hiding away something needed and then having to write more code to expose it again. Opting certain entities out of syncing requires extra effort, and it appears to support only an all-or-nothing sync, rather than say just up or down. The project was coded in Swift so unfortunately fell victim to the constant syntax changes of the language, and has been left broken on Swift 3.0. Furthermore, NSOperation is designed around returning errors rather than exceptions that Swift uses, I’m not sure how well the developer handles this but it does seem a potential area of weakness.

5th January 2017

CloudKit + Core Data + NSOperations – Syncing by Nick Harris

This is the top tutorial on Google for CloudKit syncing, but sadly has mistakes and leaves many unanswered questions. Uniquely, Nick uses a record zone per record type which obviously isn’t the right approach for these kind of records, he realises this later, and the reason is that CKRecordZoneSubscription can be scoped to a recordType. Another mistake is by subclassing CKFetchRecordsOperation to perform processing, however he strangely does then use correctly use NSOperation dependencies for the sync method. One interesting approach is to use a separate entity to store deletes, so that the real entity can be deleted as normal rather than just soft deleted; this has the advantage that predicates don’t need the additional where status != deleted.  There doesn’t appear to be any error handling if any one of the operations fails, furthermore the synchronous processing NSOperation just fatal errors if anything goes wrong with the Core Data method calls. I believe if he had access to an asynchronous NSOperationQueue class that supports errors and cancellation it would have helped a lot. Written in Swift so again falls victim to syntax changes.

5th January 2017

Apple CloudKit Engineer on Stack Overflow

A developer named farktronix gives great insight into the inner workings of CKFetchRecordZoneChangesOperation on the server. From the name of the class you might think that this returns all changes or deletes, which at first glance would appear far too much data to be useful for an initial sync, so for example, you might think instead to query to get the first set. However the developer states that the server coalesces changes before sending them (which gives the bug being discussed), this means that if a record was created and then deleted, the server is smart enough not to send either of those events in the fetch changes operation. This is a really great piece of insider info, and hopefully this user shares more in the future.

Enabling iOS 9.3 SDK in Xcode 8 for iOS 10

Some of the new iOS 10 features contain link-time checks, that is they are only enabled if the app was linked to the iOS 10 SDK. When working on an open source library it is necessary to debug when linked to an older SDK, e.g. iOS 9.3, on a newer device iOS, e.g. iOS 10. It currently isn’t possible to deploy to a newer iOS device with the old Xcode designed for the older SDK. For this reason it can be necessary to enable the older SDK in the newer Xcode, which can be done as follows:

Note: these commands assume Xcode 7 is at \Applications\Xcode.app and Xcode 8 is at \Applications\Xcode-beta.app

sudo ln -s /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk /Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS9.3.sdk

sudo ln -s /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk /Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator9.3.sdk

sudo /usr/libexec/PlistBuddy -c "Set :MinimumSDKVersion 9.3" /Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneOS.platform/Info.plist 

sudo /usr/libexec/PlistBuddy -c "Set :MinimumSDKVersion 9.3" /Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Info.plist

After running these commands, restart Xcode 8. Now you can set the Base SDK to iOS 9.3 and deploy to an iOS 10 device.

Any newer code generation features in Xcode 8 will need to be turned off, e.g. a Core Data model will need to have code generation set to none.

Welcome to my Developer Blog

I thought it would be good to start this developer blog with some photos of how my iOS development began. At university during my Ph.D. I had been building .NET Compact Framework apps in C# for Pocket PCs, usually on iPaqs with GPS sleeves and using Wi-Fi.

A bunch of iPaqs and I – 5th July 2004

Then in 2007 the iPhone came out. That summer, during my job at Microsoft as a lowly intern we bought iPhones for “research”.

Me at Microsoft – 25th July 2007

Given it was initially a US only product and SIM locked to AT&T I followed Geohot’s live blog on jailbreaking for purposes of SIM unlocking. We used Microsoft’s micro-soldering equipment to attach the wires for the hardware jailbreak, then performed the SIM unlock and then I could begin using the iPhone on a network the UK, must have been one of the first!

My first iPhone open with the jailbreaking wires attached – 25th August 2007

Microsoft was a fantastic experience but I also learned that I never wanted to work at a corporation again. So from there I began reverse engineering the frameworks and building custom apps and even distributing them via Installer that was released that same summer, and then the following year, Cydia. My first jailbreak apps were primarily around Wi-Fi scanning – my own app WiFiFoFum and a Wi-Fi positioning system for Navizon. Apple picked up on this jailbreak app phenomenon and subsequently released their public SDK and App Store, at which point WiFiFoFum was almost already to go and was one of the first apps in the store. Everything was going great with thousands of sales per day until unfortunately Apple decided to ban it for using private APIs. They still to this day do not allow Wi-Fi scanners in the store.

Since then I have been perfecting my skills, researching everything about iOS, Macs, Cocoa and Objective-C, learning all the new technologies as they become available. I develop my own apps and ones for academic research projects and the odd bit of consultancy. I’m still very interested in Wi-Fi but more recently I have developed an interest in realtime syncing of information between devices and apps for health. Hopefully you will find any information on this blog useful.