Building Real-time Collaborative Apps with Yjs and WebSockets

The modern web thrives on interactivity and, increasingly, on collaboration. From shared documents to real-time whiteboards and interactive presentations, the demand for applications that allow multiple users to work together seamlessly and simultaneously is at an all-time high. Building such applications, however, presents a unique set of challenges related to state synchronization, conflict resolution, and network latency.

Enter Yjs and WebSockets. Yjs is a powerful framework for building collaborative applications, leveraging Conflict-free Replicated Data Types (CRDTs) to handle complex state synchronization and conflict resolution automatically. WebSockets provide the real-time, bidirectional communication channel necessary to transmit updates between clients and the server instantaneously.

In this comprehensive guide, we, as senior engineers, will dive deep into the architecture, implementation, and best practices for creating robust real-time collaborative applications using Yjs and WebSockets. We’ll explore the underlying concepts, walk through practical code examples for a collaborative text editor, and discuss advanced topics like persistence, awareness, and scaling. By the end, you’ll have a solid understanding and the tools to build your own multi-user experiences.

Understanding Real-time Collaboration: The Challenges

At its core, real-time collaboration allows multiple users to view and modify the same data simultaneously, with changes propagating almost instantly to all participants. While seemingly simple, achieving this reliably and efficiently is surprisingly complex. Let’s break down the fundamental challenges:

Concurrency and Conflict Resolution

When two or more users modify the same piece of data at the same time, a conflict arises. Consider two users typing into the same text field:

• User A types “Hello”
• User B types “World” at the same position

Which change takes precedence? How do you merge them without losing data or producing inconsistent states? This is the central problem of concurrency control in collaborative systems.

Historically, two main approaches have dominated this space:

1. Operational Transformation (OT): Pioneered by systems like Google Docs, OT involves transforming operations (e.g., “insert character at index X”) before applying them, ensuring that they apply correctly to the current state, even if other operations have been interleaved. OT is incredibly powerful but notoriously complex to implement correctly due to the intricate logic required for transforming operations and maintaining a consistent history. Bugs in OT implementations are common and difficult to debug.

2. Conflict-free Replicated Data Types (CRDTs): CRDTs are data structures that can be replicated across multiple machines, allowing concurrent updates to be applied independently and then merged without conflicts. The magic of CRDTs lies in their mathematical properties: they guarantee eventual consistency, meaning all replicas will eventually converge to the same state, regardless of the order in which updates are applied. This “conflict-free” nature vastly simplifies the logic compared to OT, making them more robust and easier to implement.

Yjs embraces the CRDT approach, providing a robust and performant implementation that abstracts away the complexities of concurrent data synchronization.

Network Latency and Offline Capabilities

Real-time collaboration depends on fast communication. Network latency can cause delays in updates, leading to a perceived lag or inconsistent views among users. A good collaborative system should minimize this lag and ideally allow users to continue working even when briefly offline, syncing their changes once connectivity is restored.

CRDTs inherently support offline-first architectures. Since updates can be applied out of order and merged without conflicts, a client can queue up its changes while offline and then send them to the server (and other clients) once online, knowing that they will be correctly integrated.

Scalability and Performance

A collaborative application needs to handle a varying number of users, potentially thousands or more, within a single document or across many documents. This requires efficient network communication, optimized data structures, and a server architecture capable of scaling horizontally.

Introducing Yjs: The CRDT Powerhouse

Yjs is a high-performance, open-source JavaScript framework for building collaborative applications. It provides a shared data layer that automatically synchronizes changes across all connected clients, leveraging CRDTs under the hood.

How Yjs Works Its Magic

At the core of Yjs is the concept of a Y.Doc, which represents a shared document. Each client maintains its own local replica of the Y.Doc. When a user makes a change (e.g., types a character in a text editor), Yjs records this change as an “update” on the local Y.Doc. These updates are then broadcast to other clients (typically via a WebSocket server), who apply them to their own Y.Docs.

Because Yjs uses CRDTs, these updates can be applied in any order, and Yjs guarantees that all Y.Doc instances will eventually converge to the same consistent state. This eliminates the need for complex conflict resolution logic in your application code.

Key Features of Yjs

• CRDT-based: Automatically handles conflict resolution and ensures eventual consistency.

• Offline-first: Clients can make changes while disconnected and synchronize when reconnected.

• Extensible Data Types: Yjs provides familiar data structures that behave collaboratively:

  • Y.Text: For collaborative text editing (like a rich text editor or code editor).
  • Y.Map: Key-value store, similar to a JavaScript Map or object.
  • Y.Array: Ordered list, similar to a JavaScript Array.
  • Y.XmlFragment / Y.XmlElement: For collaborative XML/HTML structures, useful for rich text or structured documents.

• Providers: Yjs is network-agnostic. It provides “providers” (like y-websocket, y-webrtc) that handle the actual network communication, allowing you to focus on your application logic.

• Awareness: Built-in mechanism to share ephemeral, non-document-specific state, such as user cursors, selections, or online status, without affecting the document’s CRDT state.

• Pluggable Persistence: Easy to save and load the document state, enabling long-term storage.

WebSockets: The Real-time Communication Backbone

To enable the instantaneous propagation of Yjs updates, we need a persistent, bidirectional communication channel between clients and the server. This is precisely what WebSockets provide.

Why WebSockets over Traditional HTTP?

Traditional HTTP communication is stateless and request-response based. For real-time updates, you’d typically rely on techniques like:

• Polling: Clients repeatedly send requests to the server asking for updates. Inefficient and introduces lag.

• Long Polling: Clients send a request, and the server holds it open until new data is available, then responds and closes the connection. The client immediately opens a new connection. Better than polling, but still involves overhead of opening/closing connections.

WebSockets, on the other hand, establish a single, long-lived connection between the client and the server. Once established, both ends can send messages to each other at any time, without the overhead of HTTP headers or repeated connection handshakes. This makes them ideal for low-latency, high-frequency data exchange required by real-time collaborative applications.

How WebSockets Work with Yjs

When integrated with Yjs, WebSockets act as the transport layer for Yjs updates and awareness information. Here’s the typical flow:

1. Connection: A client connects to a WebSocket server, often specifying a “room” or document ID in the URL.

2. Initial Sync: Upon connection, the server sends the client the current state of the Yjs document for that room. The client applies this state to its local Y.Doc.

3. Updates: When a client makes a change to its local Y.Doc, Yjs generates an “update” (a compact binary representation of the change). This update is sent via the WebSocket to the server.

4. Broadcast: The server receives the update and broadcasts it to all other clients connected to the same room. It also applies the update to its own server-side Y.Doc replica (which can be used for persistence).

5. Application: Other clients receive the update via their WebSockets and apply it to their local Y.Docs, automatically reflecting the changes in their UI.

The y-websocket package simplifies this integration, providing both a server-side and client-side component that handles the Yjs synchronization protocol over WebSockets.

Architecting Our Collaborative Application

Before diving into code, let’s visualize the architecture of our Yjs + WebSockets collaborative application. We’ll be building a simple collaborative text editor, but the principles apply to any Yjs-powered real-time app.

High-Level Overview

The core architecture consists of multiple clients interacting with a central server via WebSockets. Each client maintains its own Yjs document instance, reflecting the shared state. The server also maintains its own Yjs document instance for each collaborative “room” or document, acting as the central hub for synchronization and potentially persistence.

Diagram in Words: The Flow of Data

Client A                                  Server                                  Client B
+-----------------+                       +-----------------------+                       +-----------------+
|   UI (textarea) |                       |   WebSocket Server    |                       |   UI (textarea) |
|      ^          |                       |          ^            |                       |          ^      |
|      |          |                       |          |            |                       |          |      |
|   Y.Text        | <--Yjs Update-->     |   Y.Doc (Room 1)      | <--Yjs Update-->     |   Y.Text        |
|      ^          |                       |          ^            |                       |          ^      |
|      |          |                       |          |            |                       |          |      |
|   Y.Doc         |                       |   Document Manager    |                       |   Y.Doc         |
|      ^          |                       |          ^            |                       |          ^      |
|      |          |                       |          |            |                       |          |      |
|   y-websocket   | <----- WebSocket ----> |   y-websocket adapter | <----- WebSocket ----> |   y-websocket   |
|   (Client)      |                       |   (Server per client) |                       |   (Client)      |
+-----------------+                       +-----------------------+                       +-----------------+
        ^                                            ^                                            ^
        |                                            |                                            |
        -------------------------------------------------------------------------------------------
                              Broadcasts Yjs Updates & Awareness to all connected clients

Explanation:

1. Client-side: Each browser tab or application instance is a client. It loads the UI (e.g., a textarea), initializes a local Yjs Y.Doc, and sets up a y-websocket client provider. This provider establishes a WebSocket connection to the server and handles sending/receiving Yjs updates and awareness messages.

2. Server-side: A Node.js application runs a WebSocket server. This server manages multiple Yjs Y.Doc instances, typically one per “room” or collaborative document. When a client connects, the server uses the y-websocket server provider to integrate the client’s WebSocket connection with the appropriate server-side Y.Doc. This provider also handles the initial synchronization and broadcasting of updates to all other connected clients for that room.

3. Data Flow (User Input):

   1. User A types into the textarea.
   2. The textarea‘s changes are bound to a Yjs Y.Text type within Client A’s local Y.Doc.
   3. Yjs detects the change and generates a compact update.
   4. The y-websocket client provider sends this update over the WebSocket to the server.
   5. The server’s y-websocket adapter receives the update and applies it to the server’s Y.Doc for that room.
   6. The server’s y-websocket adapter then broadcasts this update to all other connected clients (including Client B) in the same room.
   7. Client B’s y-websocket client provider receives the update and applies it to Client B’s local Y.Doc.
   8. Client B’s Y.Text type, being bound to its textarea, automatically updates the UI, showing User A’s changes.

4. Awareness Flow (User Cursor):

   1. User A moves their cursor in the textarea.
   2. Client A’s y-websocket provider sends an “awareness” update (containing cursor position, selection, etc.) to the server.
   3. The server broadcasts this awareness update to all other clients (including Client B).
   4. Client B receives the awareness update and displays User A’s cursor position in its UI.

This architecture decouples the complex synchronization logic (handled by Yjs) from the network transport (handled by WebSockets and y-websocket), making development much more manageable.

Setting Up the Project

Let’s create a basic project structure for our collaborative text editor. We’ll have a server-side Node.js application and a client-side HTML/JavaScript application.

Project Structure

collaboration-app/
├── server/
│   ├── package.json
│   └── server.js
└── client/
    ├── index.html
    ├── client.js
    └── package.json

Server-side Initialization

Navigate to the server/ directory and initialize a Node.js project:

mkdir server
cd server
npm init -y
npm install ws yjs y-websocket

package.json for server:

{
  "name": "collaboration-server",
  "version": "1.0.0",
  "description": "Yjs WebSocket Server",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  },
  "keywords": [],
  "author": "Khadervali",
  "license": "ISC",
  "dependencies": {
    "ws": "^8.16.0",
    "y-websocket": "^1.5.0",
    "yjs": "^13.6.14"
  }
}

Client-side Initialization

Navigate to the client/ directory and initialize a Node.js project. We’ll use a local package.json to manage client-side dependencies, even though we’ll serve client.js directly in the browser. A build step (like Webpack or Vite) would be used in a real-world app, but for simplicity, we’ll rely on browser modules.

mkdir client
cd client
npm init -y
npm install yjs y-websocket

package.json for client:

{
  "name": "collaboration-client",
  "version": "1.0.0",
  "description": "Yjs WebSocket Client",
  "main": "client.js",
  "type": "module",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "Khadervali",
  "license": "ISC",
  "dependencies": {
    "y-websocket": "^1.5.0",
    "yjs": "^13.6.14"
  }
}

Note the "type": "module" in the client’s package.json, which allows us to use ES module syntax (import ... from '...') directly in client.js.

Building the WebSocket Server

The server’s primary responsibility is to act as a central hub for Yjs documents and to relay updates between connected clients. We’ll use the ws library for the raw WebSocket server and y-websocket for Yjs integration.

server.js

// server/server.js
import { WebSocketServer } from 'ws';
import * as Y from 'yjs';
import { setupWSConnection, closePooledResources } from 'y-websocket/bin/utils';
import { readFileSync, writeFileSync, existsSync } from 'fs';
import { join } from 'path';

const PORT = process.env.PORT || 1234;
const DOCUMENTS_DIR = './documents'; // Directory to store persisted Yjs documents

// A map to store Y.Doc instances, one per room (document name)
const docs = new Map();

// Ensure the documents directory exists
if (!existsSync(DOCUMENTS_DIR)) {
  writeFileSync(DOCUMENTS_DIR, ''); // Create it.
  console.log(`Created documents directory: ${DOCUMENTS_DIR}`);
}

/**
 * Get or create a Y.Doc for a given room.
 * This function also handles persistence: loading from disk on first access,
 * and saving to disk periodically or on server shutdown.
 * @param {string} docname
 * @returns {Y.Doc}
 */
const getDoc = (docname) => {
  if (!docs.has(docname)) {
    const doc = new Y.Doc();
    docs.set(docname, doc);
    console.log(`Created new Y.Doc for room: ${docname}`);

    const docPath = join(DOCUMENTS_DIR, `${docname}.yjs`);

    // Load initial state if it exists
    if (existsSync(docPath)) {
      try {
        const initialState = readFileSync(docPath);
        Y.applyUpdate(doc, initialState);
        console.log(`Loaded initial state for ${docname} from ${docPath}`);
      } catch (error) {
        console.error(`Error loading state for ${docname}:`, error);
      }
    }

    // Save document state periodically (e.g., every 10 seconds)
    let saveTimeout = null;
    const saveDoc = () => {
      if (saveTimeout) clearTimeout(saveTimeout);
      saveTimeout = setTimeout(() => {
        const state = Y.encodeStateAsUpdate(doc);
        writeFileSync(docPath, Buffer.from(state));
        console.log(`Saved state for ${docname} to ${docPath}`);
      }, 10000); // Save every 10 seconds
    };

    doc.on('updateV2', saveDoc); // Trigger save on any document update

    // Also save on process exit
    process.on('SIGINT', () => {
      console.log(`Saving ${docname} on SIGINT...`);
      saveDoc();
      closePooledResources(); // Close any other resources Yjs might be holding
      process.exit();
    });
  }
  return docs.get(docname);
};


const wss = new WebSocketServer({ port: PORT });

wss.on('connection', (ws, req) => {
  // Extract the room name from the URL path.
  // E.g., ws://localhost:1234/my-document-1
  const url = new URL(req.url, `http://localhost:${PORT}`);
  const docname = url.pathname.substring(1); // Remove leading slash

  if (!docname) {
    console.warn('Client connected without specifying a document name. Closing connection.');
    ws.close();
    return;
  }

  console.log(`Client connected to room: ${docname}`);

  // Use y-websocket's setupWSConnection to handle Yjs sync for this connection
  // This automatically binds the WebSocket to the Y.Doc and handles updates, awareness, etc.
  setupWSConnection(ws, req, getDoc); // `getDoc` will be called with `docname`

  ws.on('close', () => {
    console.log(`Client disconnected from room: ${docname}`);
    // y-websocket's setupWSConnection handles cleanup on its own
  });

  ws.on('error', (error) => {
    console.error(`WebSocket error for room ${docname}:`, error);
  });
});

wss.on('listening', () => {
  console.log(`Yjs WebSocket server listening on ws://localhost:${PORT}`);
});

// Handle server shutdown
process.on('SIGTERM', () => {
  console.log('SIGTERM received. Closing WebSocket server...');
  wss.close(() => {
    console.log('WebSocket server closed. Saving all documents...');
    // Ensure all documents are saved before exiting
    docs.forEach((doc, docname) => {
      const docPath = join(DOCUMENTS_DIR, `${docname}.yjs`);
      const state = Y.encodeStateAsUpdate(doc);
      writeFileSync(docPath, Buffer.from(state));
      console.log(`Saved final state for ${docname} to ${docPath}`);
    });
    closePooledResources();
    process.exit(0);
  });
});

console.log('Server is starting...');

Code Explanation: server.js

• Imports: We import WebSocketServer from ws, Y from yjs, and setupWSConnection/closePooledResources from y-websocket/bin/utils. The latter provides a convenient way to integrate Yjs with a WebSocket server without implementing the Yjs sync protocol manually.

• docs Map: This map stores our Yjs Y.Doc instances. Each key represents a unique “room” or document name, and its value is the corresponding Y.Doc.

• getDoc(docname) function: This is a crucial helper.

  • It checks if a Y.Doc for the given docname already exists in the docs map. If not, it creates a new one.
  • Persistence: It attempts to load the initial state of the document from a file (.yjs extension) in the ./documents directory. If the file exists, it applies the loaded state to the new Y.Doc using Y.applyUpdate.
  • It sets up a listener for doc.on('updateV2'). Whenever the document changes (due to a client update), it triggers a debounced save operation to write the current state back to the file using Y.encodeStateAsUpdate. This ensures our document state is persistent across server restarts.
  • It also includes handlers for SIGINT (Ctrl+C) and SIGTERM to ensure documents are saved gracefully when the server shuts down.

• WebSocket Server Setup: A new WebSocketServer is created on the specified PORT.

• wss.on('connection'): When a new client connects:

  • It extracts the docname (room name) from the WebSocket URL (e.g., ws://