2.0 Migration Guide

We have heard and appreciate all of the feedback we've gotten over the past year, and it is very clear: Ditto is maturing and developers want to build more complex applications. To do that, Ditto needs to expose more of the internal functionality so that developers have more flexibility and control. Ditto 2.0 is the first step in that direction.

This is a migration guide that covers the most substantial changes that will affect most users. For a comprehensive list of all deprecated and removed methods, see the changelog.

Android permissions

On Android, permissions have changed. Remove android:maxSdkVersion="30" from ACCESS_FINE_LOCATION and ACCESS_COARSE_LOCATION.

    <uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" />    <uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION" />

Explicit types

Types can now be made explicit. Ditto 2.0 introduces DittoRegister and Counter as the first step:

Counter

• replaceWithCounter is deprecated. Instead, use DittoCounter() and increment(double)
• Counters always start at 0

Counters are always initialized with no parameters in the constructor. This is to encourage all mutations to happen within an update clause. This helps behavior be more clear -- if you want to have a mutable type, you need to mutate them using the methods available for that type.

let id = collection.upsert({  seat: '16c',  drinks: DittoCounter(),})

collection.findById(id).update(doc => {  doc.drinks.increment(1)})

DittoRegister

• Added DittoRegister to explicitly create Registers
• Added Map and Array to the list of valid register types

Map and Array are now valid register types. We call these Complex Registers. Complex Registers allow you to query and access data as their fundamental type. For example:

collection.find("array[0] == 1")collection.find("map['b'] == 'c'")
coll.findByID(docID).update({ mutDoc in  let one = mutDoc["array"][0]}

However, updating a complex register is last writer wins for the entire type, similar to a string or number. These types are useful for pulling data into Ditto from external databases, where Ditto is rarely modifying that data internally.

All of the following are valid uses of DittoRegister type:

let content: [String: Any?] = [    "string": DittoRegister(value: "string in register"),    "integer": DittoRegister(value: 123),    "float": DittoRegister(value: 4.56),    "bool": DittoRegister(value: false),    "nested": [        "inner": DittoRegister(value: "simple")    ],    "array": DittoRegister(value: [1, 2, 3]),    "complex_array": DittoRegister(value: [[[["a": 42]]]]),    "map": DittoRegister(value: ["b": "c"]),    "complex_map": DittoRegister(value: ["d": ["e": ["f": [["four": 4, "five": 5]]]]])]
let docID = try! collection.upsert(content)

All supported DittoRegister types are:

• String
• Numbers (Int, Float, etc)
• Boolean
• null
• Binary
• Map (new Complex Register)
• Array (new Complex Register)

Updating a Register

• Only use set(value) or remove() a Register. Registers cannot be partially updated.

For example, say you have a document that contains complex content that inserted from a legacy database, where all values are registers. To modify a register:

coll.findByID(docID).update(mutDoc => {  // This is ok. Retrieves the data at an index  let one = mutDoc["array"][0]
  // To update the array  let newArray = mutDoc["array"]  newArray[0] = "foo \(one)"  // To update the value  mutDoc["array"].set(DittoRegister(value: newArray))})

You can also update registers using upsert but you must not forget to always wrap your new value with DittoRegister():

coll.upsert({  "map": DittoRegister(value: newMap)})

The following example code will change the type of your register to a map. Do not do this.

coll.findByID(docID).update(mutDoc => {  // This is not ok. You should not modify a register at an index or path.  // You will lose any concurrent updates and the type will change.  mutDoc["complex_map"]["d"]["e"] = "bananas"}

HTTP API

• The HTTP API will support explicit types through the /api/v2/store/[method] endpoint.
• The /api/v1/store endpoint is deprecated, and will become unsupported in Ditto 3.0.

To create and modify a Register Array, Register Map, or Counter in the HTTP API for v2, use the /api/v2/store/[method] endpoint and annotate the value with the type you intend to use.

Endpoints

• /api/v2/store/write
• /api/v2/store/find
• /api/v2/store/findbyid

Example

A full example with curl that shows how to use the HTTP API to create explicit types. In this example, we create a friends key with a Register that is an array, and orderCount which is a counter.

We use the counter override by adding the following key to the payload:

"valueTypeOverrides": {  "orderCount": "counter",  "friends": "register"}

In v2, Arrays are registers by default. That means you do not need to add register to the valueTypeOverrides payload, but you can if you want to be explicit.

curl -X POST 'https://{app_id}.cloud.ditto.live/api/v2/store/write' \  --header 'X-DITTO-CLIENT-ID: AAAAAAAAAAAAAAAAAAAABQ==' \  --header 'Content-Type: application/json' \  --data-raw '{      "commands": [{        "method": "upsert",        "collection": "people",        "id": "123abc",        "value": {          "name": "John",          "friends": ["Susan"],          "orderCount": 5        },        "valueTypeOverrides": {          "orderCount": "counter"        }      }]  }'

To find this document you can use /api/v2/store/findbyid:

curl --location --request POST '483acae0-035a-43f2-afa9-2d28239ad721.cloud.ditto.live/api/v2/store/findbyid' \--header 'X-DITTO-CLIENT-ID: AAAAAAAAAAAAAAAAAAAABQ==' \--header 'Content-Type: application/json' \--data-raw '{  "collection": "people",  "id": "123abc"}'

{  "collection": "people",  "id": "123abc"}

Optionally, When you query for this data using /api/v2/store/find, you can use the key serializedAs: latestValuesAndTypes to receive a response with each type specified:

Request:

curl -X POST 'https://{app_id}.cloud.ditto.live/api/v2/store/find' \  --header 'X-DITTO-CLIENT-ID: AAAAAAAAAAAAAAAAAAAABQ==' \  --header 'Content-Type: application/json' \  --data-raw '{    "collection": "people",    "query": "name=='John'",    "limit": 2,    "serializedAs": "latestValuesAndTypes"}'

Response:

{  "value": {    "_id": "123abc",    "fields": {      "name": {          "register": { "value": "John" },      },      "friends": {          "register": { "value": ["Susan"] },      },      "orderCount": {          "counter": { "value": 5 }      },    }  }}

CDC

This feature is only available on Dedicated Clusters

If you're using Ditto's CDC (Change Data Capture), such as a Kafka connector, you can upgrade to a v2 topic to start getting information about explicit types. For example, if you create a DittoRegister using Ditto v2, you'll get a type annotation similar to the HTTP response. Contact your support engineer for more information.

Growable arrays

• The Growable Array is deprecated.
• .push() and .pop() and array[index].remove() are deprecated.

RGAs will be entirely unsupported in Ditto 3.0, so it is important that you start migrating away from using Growable Arrays and start using a DittoRegister or a Map instead.

We may bring back the Growable Array in 2023 for text editing. However, we do not see the demand for Growable Arrays right now, so we decided to deprecate it for the time being. Please reach out through the help center on the bottom right of your screen if you are interested in Growable Arrays. We would love to hear about your use case.

insert

insert and insertWithStrategy have been removed from the Collection type. You should instead use upsert or upsert_with_strategy.

If you were previously providing a document identifier to an insert call then you should instead provide it to the upsert call by specifying it under an _id key at the root of the value passed to the upsert call.

Write strategy

• Ditto 2.0 will remove support for the overwrite write strategy.

Sync limits

In 1.x findAll().limit(10) would not limit replication by default. This would cause small peers to crash if they accidentally pulled down more data from the Big Peer than their platform or hardware could handle.

veryLargeCollection.findAll().limit(10).observe(callback) // CRASH!!!!

In the 2.0, limit(10) will only replicate 10 documents as expected.

veryLargeCollection.findAll().limit(10).observe(callback) // OK

Forward-compatibility with concurrent types

• It is valid to use multiple types concurrently at the same document path.

Distributed applications typically only add new fields and never remove or modify existing fields in production. Despite these validation code paths, bugs happen. It could be possible to have a version of your application that changed the type of a field that an older application version is using. This would break the forward-compatibility of your software. This could cause a crash in production if an application version is expecting a different shape to your data.

If this happens, Ditto 2.0's concurrent types ensure that data is never lost and that old code can still work with new data. Devices can always access all types that have been added to a particular field. Ditto does its best to preserve data, even if the type was changed by another, incompatible version of your application. This feature allows you to more easily build forward-compatibile applications.

It is another tool that Ditto gives you so that you can manage schema changes more robustly, and reduce the chance of data loss or crashes in production.

Device A

try! coll.upsert(["edited_by": "john"])

Device B

try! coll.upsert(["edited_by": ["timestamp": 16827219234, "user_id": "abc123"])

After synchronization

const doc = coll.findByID(docID)doc["edited_by"].value // latest timestamp: ["timestamp": ..., "user_id": ...]doc["edited_by"].dictionaryValue // ["timestamp": ..., "user_id": ...]doc["edited_by"].stringValue // "john"

JavaScript changes

The JS document API changes substantially with Ditto v2. In v1, we've proxied instances of Document and MutableDocument allowing us to make accessing and updating contents of a document feel as simple as manipulating a regular JavaScript object.

This "magic", however, led to many edge cases, inconsistencies, and confusion. Together with the introduction of explicit CRDT types in v2, we've taken the opportunity and redesigned the JS document API to be more explict and align with the APIs of the other Ditto SDKs.

Accessing document content

To access and update the contents of a document, you'll now mainly interact with DocumentPath and MutableDocumentPath instances directly. Given a document, you can get the corresponding path object via the .path property:

const collection = ditto.store.collection('abc')const document = await collection.findByID('123')const path = await document.path

This path instance represents the document content at the root. With that, you can access a property at a specific key-path via the at() method, which yields another path instance representing the document content at that key-path:

// Both are equivalent, the latter is a little more efficient:const pathDeepDown1 = document.path.at('deep').at('down')const pathDeepDown2 = document.path.at('deep.down')

Since this is so common, Document and MutableDocument offer a convenience method at(), which is equivalent to path.at(), so the above becomes:

const pathDeepDown3 = document.at('deep.down')

With that, we can now access the value of a property at a given key-path, regardless of the underlying CRDT type:

const name = document.at('deep.down.name').value

Or the value for a specific CRDT type:

const name = document.at('deep.down.name').register.valueconst count = document.at('deep.down.count').counter.valueconst count = document.at('deep.down.elements').rga.value// ...

Updating document content

There are only 2 update operations accessible via a MutableDocumentPath object: set() & remove(). set() allows you to create a property with a specific CRDT type, while remove() allows you to remove that property:

await collection.findByID('123').update((mutableDocument) => {  mutableDocument.at('some.new.name').set(new Register(''))  mutableDocument.at('some.existing.property').remove()})

All other mutations, i.e. updating the value of an existing property, must be performed via the operations offered by the corresponding (mutable) CRDT type:

await collection.findByID('123').update((mutableDocument) => {  mutableDocument.at('deep.down.name').register.set('Peter Pan')  mutableDocument.at('deep.down.count').counter.increment(456)
  mutableDocument.at('deep.down.elements').rga.insertAt('some-element-789', 1)  mutableDocument.at('deep.down.elements').rga.removeAt(0)  mutableDocument.at('deep.down.elements').rga.push('some-element-abc')  const last = mutableDocument.at('deep.down.elements').rga.pop()  // ...})