CLEMA protocol

Synchronization

Durable changes are events in ordered streams. A device follows the streams that concern its user, applies what they carry and remembers how far it got, so it catches up after any time away without reloading what it has.

Streams

Every conversation — direct, group or channel — has a conversation stream of the events all its current members see. An event is written to it once, however many members there are. Every account has an account stream of what only its user sees: which conversations they are in, their own state, and events meant for a conversation's owner and admins rather than for all its members.

A conversation event reaches the members who joined the conversation before it was written, live as in a pull. Every other event — presence, typing, session, phone, transport and direct-transfer events — is a real-time signal and belongs to no stream.

Each stream numbers its events with seq, from 1, and never reuses a number. A live event and a mutation response carry their place in the envelope: the stream is the one of the conversation named by conversation_id, and without it the account stream:

{"type":"message.created","conversation_id":"c_38a2f37d9a17e9230d113bb0f8931b2a","seq":"126","data":{...}}
{"type":"draft.updated","seq":"42","data":{...}}

A mutation response carries the place of the event its caller's devices receive for it; a retry carries the original place. The events of a sync.pull page belong to the stream that was pulled and carry no place of their own beyond their seq.

Streams converge rather than record history. An event is shaped for its recipient when it is delivered: a message event carries the message as the recipient sees it at that moment. An event about something the recipient can no longer see — a message they hid or cleared, a conversation they left — keeps its identifiers and what it says itself, such as the scope of a deletion, and nothing more. Hiding and clearing are final on a device: a later event about a message the user hid, or cleared through, does not bring it back. Clients upsert snapshots by msg_id and never rebuild past revisions from event bodies.

A conversation snapshot inside an event carries only the groups of fields its stream changes, each group whole, and the client replaces each group it carries in the conversation it holds:

conversation.created carries the whole conversation as it stands when the event is delivered, like conversation.get; one the user has lost again since carries only its conversation_id, and its conversation.deleted follows. Unread counts hold as of the snapshot's seq: a device that has applied later events of the conversation adds the incoming messages among them.

Members joining and leaving a channel are not events of its stream, which its subscribers follow: the member_count of a channel's snapshot holds as of that snapshot, and a client refreshes it with conversation.get when it shows it. A member whose own role changes reloads what the new role lets them see of the conversation — its members, invitations and bans — and forgets what it no longer does.

Following Streams

A device keeps a cursor for its account stream and one for every conversation it follows, and moves a cursor only past events it has applied. An event — live, or named by a mutation response — is:

• the next one when its seq is one past the cursor: the device applies it and moves the cursor;
• a duplicate when its seq is at or below the cursor, and is ignored;
• behind a gap when its seq is further ahead: the device pulls the stream from its cursor first.

An event of a type the device does not know is applied as nothing: the device passes over what it says, but not its place, and moves its cursor past it.

A device follows the conversations its account stream gives it: conversation.created starts one, and conversation.deleted ends it, as does not_found from a pull or from conversation.get.

To start following a conversation, a device takes a snapshot of it — from conversation.created, conversation.list or conversation.get — and follows its stream after the snapshot's seq. Messages at or below that seq are not replayed: the device reads what it needs of them from history, at least the unread ones, with message.list after the snapshot's read_through. When a conversation starts because something arrived in it — its first message, or the user joining — its unread messages count as newly arrived, and the device shows and notifies them as usual. Events of the conversation that arrive while it starts are kept and then handled by the rules above. An event of a conversation the device does not follow yet starts it the same way, with conversation.get. A snapshot that arrives after the device let the conversation go, such as the answer to a conversation.get sent before, is ignored.

A device that follows nothing yet, such as one just signed in, asks sync.state where its account stream stands, loads conversation.list, contact.list, block.list, privacy.get, pack.list and its own profile.get, and then follows the account stream from the position sync.state gave and every listed conversation from its snapshot. It replays no history and notifies nothing.

After reconnecting, a device catches up:

1. It asks sync.state where its streams stand.
2. It pulls its account stream if the head differs from its cursor.
3. It pulls every conversation it follows whose head differs from its cursor, and starts every listed conversation it does not follow. A conversation it follows that the answer leaves out is let go, unless the device started following it after sending the request.
4. It asks pack.list where the smiley packs stand.

A head below the cursor is pulled too: the stream went back — a restored database, or writes lost in a crash — and the pull resets the cursor. A device keeps at most eight requests of a catch-up in flight at once.

Where Streams Stand

{"id":"state-1","type":"sync.state"}

{"id":"state-1","type":"sync.state","ok":true,"data":{"account":"41","conversations":{"c_38a2f37d9a17e9230d113bb0f8931b2a":"125","direct:12345:67890":"46"}}}

A head is the last seq a stream has given out, whether or not its event is still kept; 0 before the first.

Pulling a Stream

{"id":"sync-1","type":"sync.pull","data":{"conversation_id":"c_38a2f37d9a17e9230d113bb0f8931b2a","after":"124","limit":100}}

{
  "id":"sync-1","type":"sync.pull","ok":true,
  "data":{
    "events":[
      {"seq":"125","type":"message.edited","data":{"msg_id":"12410","conversation_id":"c_38a2f37d9a17e9230d113bb0f8931b2a","from":67890,"kind":"text","payload":{"text":"Corrected"},"time":"2024-01-15T14:30:00Z","revision":2,"edited_at":"2024-01-15T14:32:00Z","reactions":[]},"time":"2024-01-15T14:32:00Z"}
    ],
    "next":"125",
    "has_more":false
  }
}

Apply a whole page before persisting next, and pull again while has_more is true. Pulling the stream of a conversation the user does not belong to returns not_found.

Retention and Reset

A stream keeps its events for at least 30 days, as long as an unused session lives. A cursor that cannot be followed is reset: one below the events a stream still keeps, one beyond its head, and, for a conversation, one from before the user last joined it. A server may also reset a cursor so far behind that starting again costs less than catching up. A server whose database is restored from a backup resets every cursor from before the restore: devices may hold events it lost, at positions it would hand out again. The answer carries no events, reset: true and next at the stream's head:

{"id":"sync-2","type":"sync.pull","ok":true,"data":{"events":[],"next":"9120","has_more":false,"reset":true}}

The device then starts the stream again. A conversation it starts following anew, reloading whatever of it the device keeps — pins, members, bans, history. The account stream it starts like a device that follows nothing yet, keeping the conversations it follows that conversation.list still returns.