There is some confusion on the internet about our syncing solution in Instacast 3. Some think we are still using iCloud and confuse "Cloud Sync" with "iCloud Sync". Let me know on Twitter if you have a suggestion for a better name. Let me get this straight, we are NOT using iCloud anymore. There are numerous reasons, why we're doing this, a lot of them I explained extensively on this blog and on our support pages before. I don't want to recap those today. Instead I like to give you an overview, an insight if you will, into our newly built cloud syncing solution.
I teamed up with Stefan of Mediaatelier on this, who came up with the idea of an algorithms that could be used to synchronize an object hierarchy between multiple devices without building up a lot of conflicts. This algorithm has been implemented, tested and revised over 4 months to make up the solution that it is today. On top of that we built a small server infrastructure that could be scaled to millions of users, if it ever becomes necessary. We now have a generic synching solution that can be used for all kinds of applications and we are currently testing it first with Instacast 3 an application that itself has a couple of thousand users. The start has been rocky so far and we're working hard on ironing all the kinks out. Pretty late in the development process we were testing iCloud as server infrastructure for a last time, but we simply couldn't get it to work. iCloud wasn't either fast nor was it reliable enough. iCloud today is simply not an option for us.
So how does it work in Instacast? We are using simple WebDav servers as our backend. This makes scaling and setting up new server quite easy and also guarantees the necessary speed as long as servers are not overloading, since WebDav is as simple as making special HTTP requests, something that is very optimized today. WebDav however has no concept of pushing data to your devices. Devices have to request data from the server from time to time. Because this polling is not a good experience for the user and a refresh button looks so outdated, we came up with a couple of mechanisms that give the user the impression that the data is being pushed.
The first thing the sync does, is, it sends out small UDP broadcast packets at a frequency of 1Hz (nerds know what that means) around the local WiFi network along with some basic state information. Using the data in those packets a device can tell if a data upload happened on another device and thus can request the download of that data accordingly. This is very fast and happens normally within 1 second. There can be a number of problems however that prevent those packets from being distributed with the network, and because this method only works within the local network, we also have a similar mechanism that works with the Key-Value store of iCloud. That means, if you have an iCloud account configured, your device can upload it's state information into iCloud also via the mobile network and iCloud pushes this state information to all your other devices that do not have to be connected to the same network. A simple internet connection is sufficient. Because that can fail as well, the application is also polling every time itself makes a change or as soon as the user make the app visible on screen. The sum of these mechanisms make up our sync solution and we think it works pretty great so far.
If you are interested in learning more or in using the framework and our cloud solution for your own projects, please let us know.
December 9th, 2012 • Permalink
04/01/2015 - Snowflake Lite on the Mac App Store
03/02/2015 - Announcing Snowflake 1.0
12/10/2014 - Instacast for Mac 2.0 Now Available
10/31/2014 - Up Next in Instacast 5
10/22/2014 - Instacast 5 available today
09/16/2014 - Updates on Instacast 5
06/27/2014 - Instacast Cloud Memberships
06/06/2014 - Instacast 4.6 and Instacast Cloud
03/26/2014 - Sharebox adds Drag & Drop sharing to the Dropbox menubar icon
01/22/2014 - Instacast 4.2 adds News Mode
01/12/2014 - Hello 2014