State-based Synchronization

Sung Cho Dec 10, 2018 Product Update

Synchronization of your notes and books is now faster and more stable from v0.4.9 release. The reason is that Dnote synchronization is no longer based on transaction log but instead on the server’s state. With this change, Dnote will provide you with a more reliable sync experience and move one step further to its long-term goal of protecting your privacy.

Prior to v0.4.9 release, Dnote sync worked by keeping a log of all user actions that ever happened. Namely, those actions were add_book, edit_book, delete_book, add_note, edit_note, and remove_note.

Concretely, here is how the old synchronization system worked. Whenever a user performed one of those actions CLI saved a log locally along with a timestamp. Later, when dnote sync ran, the action log would be sent to the server and server would process, save the actions. Finally, the server would return any actions that the CLI did not know about, based on the bookmark that the CLI sent. CLI would then process the actions returned from the server and save the new bookmark.

This old synchronization strategy was inspired by message queues such as Apache Kafka or Redis Stream. For instance, Kafka server keeps track of a stream of events and the clients individually keeps track of the ID of the event that they have read up to.

...
987 `add_book` `{"label": "javascript"}`
988 `add_note` `{"book_label": "javascript", "content": "hello world!"}`
989 `add_note` `{"book_label": "javascript", "content": "foo"}`
990 `remove_note` `{"uuid": "7ef04fb6-cf01-473a-94a2-0fd75be0307e"}`
...

Given the log above, the CLI used to add new entries to it and kept track of the id of the furthest action that ingested from the server in the last sync.

However, the log-based synchronization was not without edge cases. The main reason was that there were some complicated dependencies among the actions. For instance, if the server recorded remove_note and the client later sent edit_note for the same note, the server needs to ignore the edit_note. Or if the server gets add_book and remove_book for the same book in order, add_book action should be invalidated. These logical relationships between actions created a complicated graph of dependencies and blew up the complexity.

With state-based synchronization, the server and the CLI no longer keep track of the action log. Instead, Dnote system treats the server’s state as the source of truth and replicates it to an arbitrary number of clients while resolving conflicts.

When dnote sync runs, a CLI pulls down all changes that it has not seen yet and then uploads all changes the server hasn’t seen yet. The system can determine the unseen changes based on a monotonously increasing sequence of integers attached to all Dnote resources. As in the Apache Kafka model, the client still keeps track of the furthest number it has seen thus far as a bookmark.

▶ dnote sync
  • resolving delta (total 17). done. # receives the server changes
  • sending changes (total 5). done. # sends the client changes to the server
  ✔ success

# saves the furthest resource number in the bookmark

To recover from possible erroneous states, a ‘full sync’ was also added. It can be invoked by running dnote sync -f or dnote sync --full. The full sync will receive all resources from the server from the beginning of time, clean up any problematic states in the CLI, and send local changes to the server as needed.

I am very proud to introduce this new synchronization feature. It will provide you with a more smooth way of synchronizing your Dnotes across all your machines. Thanks for your continued support.

Get Dnote