Syncing

How Does Ditto Sync Data?

In this narrated video, take a glimpse under the hood of Ditto's unique sync process. This video will explain how Ditto utilizes multiple transports at once to efficiently sync data with or even without the internet, as well as the strengths and limitations of each transport that come together to form what we call the Rainbow Connection.

By default, Ditto will not sync data with other devices. When the device is observing a query, it requests to synchronize data that matches the query from other devices in the mesh network.

In other words, when a live query is created, Ditto's sync system pulls data from other devices. There is no way to "push" data through the API explicitly. Devices select data they are interested in through the query system and then synchronize with the mesh network based on that query.

Multi-hop replication

Given that Ditto works peer-to-peer, devices can form into arbitrary groups based on the proximity to one another, or rather they create an ad-hoc mesh network. Ditto's sync system allows for devices to share data through another device, called "multi-hop" sync. The only requirement for this to occur is that all devices in the chain must be observing the same data, as shown below:

Sync

The small peer uses a selfish, query-based synchronization protocol. That means each device needs to explicitly tell other peers what data to sync using a query. Each peer also must explicitly opt-in to start synchronizing data.

Preferably, you should tell ditto to start synchronizing early on in your application's life cycle like in your AppDelegate.application(_:didFinishLaunchingWithOptions:) or Application.onCreate method. Your application only needs to call this function once.

try! ditto.startSync()

Subscribe

Creating a subscription acts as a signal to other peers that you are interested in receiving updates when local or remote changes are made to documents that match the given query.

The returned Subscription object must be kept in scope for as long as you want to keep receiving updates. This function is useful if you're interested in subscribing to changes on a query to have a local offline copy of the data.

// Register live query to update UIlet subscription = ditto.store.collection("cars").find("color == 'red'").subscribe()

• Call .subscribe after ditto.startSync() to synchronize and download real-time data from other peers.
• To stop the subscription, call subscription.cancel().

Observe

Create a query to get notified when the database gets new changes. The returned LiveQuery object must be kept in scope for as long as you want to keep receiving updates. Learn more about how to create queries

// --- Action somewhere in your applicationfunc userDidInsertCar() {    _ = try? ditto.store.collection("cars").upsert([        "model": "Ford",        "color": "black"    ] as [String: Any?])}
// Register live query to update UIlet liveQuery = ditto.store.collection("cars").find("color == 'red'")    .observeLocal { cars, event in        // do something    }

• Call .observeLocal to observe real-time data from your local database.
• The callback will immediately run for documents that fit the query.
• The callback will get called with all local changes that fit the query.
• The callback will get called for each sync or write transaction.

LiveQueryEvent

info

Reactive frameworks like React, Jetpack Compose, and SwiftUI are designed to make UI development more efficient and help programmers build responsive applications with minimal effort. They achieve this by abstracting the process of updating the UI and providing optimized solutions for managing state and rendering components. This eliminates the need for programmers to manually implement a diffing algorithm.

If you are not using a reactive framework such as React, Jetpack Compose, or SwiftUI, you might benefit from reacting to changes from Ditto's LiveQueryEvent object. A LiveQuery callback has two arguments: docs and event. The event argument allows you to identify which documents have changed in the database. There are two types of events:

• Initial: The first event that will be delivered and it will only be delivered once.
• Update: This event will be delivered each time the results of the provided query change. It contains information about the set of documents that previously matched the query before the update, along with information about what documents have been inserted, deleted, updated, or moved, as part of the set of matching documents.

For more information, see LiveQueryEvent in the Ditto API reference for your language.

How can I subscribe to multiple queries?

Option A. Store them as multiple properties on a managed object that is kept in scope. If the subscription falls out of scope, it will become garbage collected and sync will stop.

var subscribeA: DittoSubscription? = nilvar subscribeB: DittoSubscription? = nil
var liveQueryA: DittoLiveQuery? = nilvar liveQueryB: DittoLiveQuery? = nil

Option B: You could also store these objects as arrays.

var subscriptions = [DittoSubscription]()var liveQueries = [DittoLiveQuery]()

How can I stop to subscribing data?

Ensure that you always cancel or stop on subscriptions when you no longer wish to subscribe to a query.

// subscription = nil // this does not worksub.cancel() // this is correct!

// subscriptions.removeAll() Do not do this <- this will not work
for sub in subscriptions {   sub.cancel()}
subscriptions.removeAll() // After calling cancel, it is safe to remove them from the array.

Examples

1: Time-based syncing

One common design pattern is to only synchronize data that was written within the past 24 hours. At the end of that 24 hours, documents older than 24 hours are evicted and a new query is created.

Let's look through a typical example for syncing data: four flight attendants walk through an airlane and record passenger meal orders on their tablets.

The database has a flights collection, with each flight represented as a document. Each document has a createdAt timestamp.

{  "createdAt": "2022-09-17T20:00:46.945Z",  "flightNo": "DIT101",  "orders": {    "abcdef123": {      ...    }  }}

We want to only store data on the device that was created in the past 24 hours. Anything older than 24 hours should be removed locally from the device. However, we don't want to remove this data from the entire mesh network, because we may want to do analytics on it later using the Big Peer. So, we do not use remove() because that deletes data from the entire network, including the Cloud. We use evict() to remove data from the local device because the data is not relevant anymore.

ditto.store.collection("flights").find("createdAt <= $args.yesterday", args: [  "yesterday": yesterday(),]).evict()

One way to implement eviction would be to create a global interval. For example, every 24 hours check to see if there is any data on the local machine that is irrelevant. If so, evict it.

Whenever you have an eviction query that changes over time it is important that you also properly stop your subscription and update it with a new query each time your eviction query changes. This is because if you try to evict data that you are also subscribed to, then after you evict the data you will immediately sync it back to the device since the subscription is still syncing that data.

Using the example of evicting data every 24 hours we will see how to stop the subscription, evict the data, and then start the new subscription.

// 24 hours pass. Need to evict irrelevant data.
// 1. Stop subscriptionsubscription.cancel()
// 2. Evict irrelevant dataditto.store.collection("flights").find("createdAt <= $args.yesterday", args: [  "yesterday": yesterday(),]).evict()
// 3. Start new subscriptionsubscription = ditto.store.collection("flights").find("createdAt > $args.yesterday", args: [                 "yesterday": yesterday(),               ]).subscribe()

Notice that the query for eviction is exactly the opposite as the query for subscription. These queries should not overlap, so you must stop your subscription before you evict. Your subscription variable should remain in scope throughout the runtime time of the app, so that it will continue to sync data and not be garbage collected.

2. Stateful syncing

Another common design pattern is to build state into each document type. In an ordering application, orders go through a lifecycle.

OPEN -> IN PROGRESS -> COMPLETE -> FILLED
| -> CANCELLED

When the application starts, the device only wants to synchronize orders that are not filled or cancelled, and evict any orders that have been completed.

self.query =  self.ditto.collection("orders")  .find("status != 'COMPLETE'")  .subscribe()self.ditto.collection("orders").find("status == 'COMPLETE'").evict()

Once an order has been completed, document as “FILLED”. Live queries will fire and update the front-page.

self.ditto.collection("orders").upsert([  "_id": id,  "status": "COMPLETE"])