Distributed

Distributed replication

Replicate a SQLite database across nodes over gRPC, bootstrap new replicas automatically, choose write concerns, and fail over through etcd.

Table of Contents

Sirannon replicates a SQLite database across multiple nodes with change propagation, new-node bootstrapping, write concerns, and coordinator-backed failover. Each node owns its own database file; Sirannon moves checksummed change batches between them, so nothing depends on a shared network filesystem.

The production path is primary-replica: one primary accepts writes, replicas serve reads and can forward writes, and coordinator mode manages authority when failover is enabled. Every change carries a Hybrid Logical Clock timestamp for deterministic causal ordering across nodes. This page builds two files, primary.ts and replica.ts. Each step highlights exactly what changed as the node comes together.

Start the primary

Replication traffic runs over its own transport. Production node-to-node replication uses gRPC with mutual TLS, and an in-memory transport serves tests. Open the primary's database file, take a writer connection that the engine applies changes through, create a ChangeTracker for the ordered change log, point a gRPC transport at the local interface, and pass them to a ReplicationEngine in the primary role.

primary.ts
import { ChangeTracker, Sirannon } from '@delali/sirannon-db'
import { betterSqlite3 } from '@delali/sirannon-db/driver/better-sqlite3'
import { ReplicationEngine, PrimaryReplicaTopology } from '@delali/sirannon-db/replication'
import { GrpcReplicationTransport } from '@delali/sirannon-db/transport/grpc'
 
const dbPath = './data/orders.db'
const driver = betterSqlite3()
const sirannon = new Sirannon({ driver })
const db = await sirannon.open('orders', dbPath)
const writerConn = await driver.open(dbPath)
const tracker = new ChangeTracker({ replication: true })
 
const transport = new GrpcReplicationTransport({
  host: '0.0.0.0',
  port: 4200,
  tlsCert: './certs/primary.crt',
  tlsKey: './certs/primary.key',
  tlsCaCert: './certs/ca.crt',
})
 
const engine = new ReplicationEngine(db, writerConn, {
  nodeId: 'primary-us-east-1',
  topology: new PrimaryReplicaTopology('primary'),
  transport,
  snapshotConnectionFactory: () => driver.open(dbPath, { readonly: true }),
  changeTracker: tracker,
})
 
await engine.start()
 
await engine.execute('CREATE TABLE IF NOT EXISTS accounts (id INTEGER PRIMARY KEY, holder TEXT NOT NULL, balance INTEGER NOT NULL DEFAULT 0)')
await engine.execute('CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, total INTEGER NOT NULL)')
 
await engine.executeBatch('INSERT INTO accounts (id, holder, balance) VALUES (?, ?, ?)', [
  [1, 'Emma Wright', 5000],
  [2, 'Kenji Tanaka', 2000],
])

The engine's second argument is a writer connection that the engine applies incoming changes through, opened here with driver.open(dbPath). The snapshotConnectionFactory opens a read-only connection so a joining node can stream a consistent snapshot without blocking live writes, and changeTracker records the ordered change log the engine sends to replicas. The two CREATE TABLE statements and the seed define what the primary owns: accounts and orders are the two tables a replica copies during its first sync.

Point a replica at the primary

The replica is its own file. It opens its own database, takes its own writer connection and change tracker, builds a transport in the same way, and takes the replica role. Two options differ from the primary: transportConfig.endpoints names the primary to pull from, and writeForwarding lets the replica accept a write and forward it upstream. The primary.ts tab still holds the file from the previous step; the replica.ts tab is the new file you add now.

import { ChangeTracker, Sirannon } from '@delali/sirannon-db'
import { betterSqlite3 } from '@delali/sirannon-db/driver/better-sqlite3'
import { ReplicationEngine, PrimaryReplicaTopology } from '@delali/sirannon-db/replication'
import { GrpcReplicationTransport } from '@delali/sirannon-db/transport/grpc'
 
const replicaPath = './data/orders-replica.db'
const driver = betterSqlite3()
const sirannon = new Sirannon({ driver })
const replicaDb = await sirannon.open('orders', replicaPath)
const replicaConn = await driver.open(replicaPath)
const replicaTracker = new ChangeTracker({ replication: true })
 
const replicaTransport = new GrpcReplicationTransport({
  host: '0.0.0.0',
  port: 4200,
  tlsCert: './certs/replica.crt',
  tlsKey: './certs/replica.key',
  tlsCaCert: './certs/ca.crt',
})
 
const replicaEngine = new ReplicationEngine(replicaDb, replicaConn, {
  nodeId: 'replica-eu-west-1',
  topology: new PrimaryReplicaTopology('replica'),
  transport: replicaTransport,
  transportConfig: { endpoints: ['primary.example.com:4200'] },
  writeForwarding: true,
  changeTracker: replicaTracker,
})
 
await replicaEngine.start()

First sync

A new node copies the whole dataset before it starts applying incremental changes. The source streams the schema and table data to it in checksummed batches, then sends a manifest of row counts and primary-key hashes. The new node checks each batch against that manifest, catches up on any writes that arrived while it was copying, and starts serving reads once its replication lag falls below the configured threshold. Its sync state advances through four phases, which you read from phase on engine.status().syncState: pending before it picks a source, syncing while it copies tables, catching-up while it applies those buffered writes, and ready once it has caught up. A replica refuses reads until it reaches ready.

That await replicaEngine.start() call from the previous step begins the join. In the replica.ts tab, keep polling the sync state until the phase reaches ready:

import { ChangeTracker, Sirannon } from '@delali/sirannon-db'
import { betterSqlite3 } from '@delali/sirannon-db/driver/better-sqlite3'
import { ReplicationEngine, PrimaryReplicaTopology } from '@delali/sirannon-db/replication'
import { GrpcReplicationTransport } from '@delali/sirannon-db/transport/grpc'
 
const replicaPath = './data/orders-replica.db'
const driver = betterSqlite3()
const sirannon = new Sirannon({ driver })
const replicaDb = await sirannon.open('orders', replicaPath)
const replicaConn = await driver.open(replicaPath)
const replicaTracker = new ChangeTracker({ replication: true })
 
const replicaTransport = new GrpcReplicationTransport({
  host: '0.0.0.0',
  port: 4200,
  tlsCert: './certs/replica.crt',
  tlsKey: './certs/replica.key',
  tlsCaCert: './certs/ca.crt',
})
 
const replicaEngine = new ReplicationEngine(replicaDb, replicaConn, {
  nodeId: 'replica-eu-west-1',
  topology: new PrimaryReplicaTopology('replica'),
  transport: replicaTransport,
  transportConfig: { endpoints: ['primary.example.com:4200'] },
  writeForwarding: true,
  changeTracker: replicaTracker,
})
 
await replicaEngine.start()
 
for (;;) { 
  const { phase, completedTables, totalTables } = replicaEngine.status().syncState
  console.log(`sync ${phase}: ${completedTables.length}/${totalTables} tables copied`)
  if (phase === 'ready') break
  await new Promise((resolve) => setTimeout(resolve, 500))
}

The loop prints the phase as each table finishes copying:

sync pending: 0/0 tables copied
sync syncing: 1/2 tables copied
sync syncing: 2/2 tables copied
sync catching-up: 2/2 tables copied
sync ready: 2/2 tables copied

Read the same syncState object directly to see the detail behind those counts. During the copy it reports the source, the tables finished so far, and the total to expect. snapshotSeq stays null until the copy completes and catch-up begins, and startedAt records when the sync began as a Unix timestamp in milliseconds. The snapshots below come from one sample run of a replica partway through its first sync, so the startedAt timestamp and the snapshotSeq sequence differ on each run. They are printed with console.log because snapshotSeq is a bigint that JSON.stringify cannot serialise.

const { syncState } = replicaEngine.status()
 
console.log(syncState)

Once catch-up finishes, the phase reaches ready, snapshotSeq holds the sequence the snapshot was taken at, and the replica is safe to read from.

const { syncState } = replicaEngine.status()
 
console.log(syncState)
 
if (syncState.phase === 'ready') {
  const orders = await replicaEngine.query('SELECT id, total FROM orders')
}

For databases too large to transfer over the network, copy the SQLite file out of band, then start the replica from a known sequence so it skips the copy and follows changes from that point on. Pass initialSync: false and resumeFromSeq in the replica's ReplicationEngine config.

replica.ts
const replicaEngine = new ReplicationEngine(replicaDb, replicaConn, {
  nodeId: 'replica-eu-west-1',
  topology: new PrimaryReplicaTopology('replica'),
  transport: replicaTransport,
  transportConfig: { endpoints: ['primary.example.com:4200'] },
  writeForwarding: true,
  changeTracker: replicaTracker,
  initialSync: false, 
  resumeFromSeq: 48210n,
})

Write concerns

Writes go through the engine so it can count acknowledgements before returning. Control how many replicas must acknowledge a write with writeConcern on engine.execute.

import { ChangeTracker, Sirannon } from '@delali/sirannon-db'
import { betterSqlite3 } from '@delali/sirannon-db/driver/better-sqlite3'
import { ReplicationEngine, PrimaryReplicaTopology } from '@delali/sirannon-db/replication'
import { GrpcReplicationTransport } from '@delali/sirannon-db/transport/grpc'
 
const dbPath = './data/orders.db'
const driver = betterSqlite3()
const sirannon = new Sirannon({ driver })
const db = await sirannon.open('orders', dbPath)
const writerConn = await driver.open(dbPath)
const tracker = new ChangeTracker({ replication: true })
 
const transport = new GrpcReplicationTransport({
  host: '0.0.0.0',
  port: 4200,
  tlsCert: './certs/primary.crt',
  tlsKey: './certs/primary.key',
  tlsCaCert: './certs/ca.crt',
})
 
const engine = new ReplicationEngine(db, writerConn, {
  nodeId: 'primary-us-east-1',
  topology: new PrimaryReplicaTopology('primary'),
  transport,
  snapshotConnectionFactory: () => driver.open(dbPath, { readonly: true }),
  changeTracker: tracker,
})
 
await engine.start()
 
await engine.execute('CREATE TABLE IF NOT EXISTS accounts (id INTEGER PRIMARY KEY, holder TEXT NOT NULL, balance INTEGER NOT NULL DEFAULT 0)')
await engine.execute('CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, total INTEGER NOT NULL)')
 
await engine.executeBatch('INSERT INTO accounts (id, holder, balance) VALUES (?, ?, ?)', [
  [1, 'Emma Wright', 5000],
  [2, 'Kenji Tanaka', 2000],
])
 
await engine.execute( 
  'INSERT INTO orders (id, total) VALUES (?, ?)',
  [1, 4999],
  { writeConcern: { level: 'majority', timeoutMs: 5000 } },
)

The levels are 'local', 'majority', and 'all'. In static mode, omitting writeConcern returns after the local commit. In coordinator mode, omitting it selects 'majority', and a successful majority write survives automatic primary failover when an eligible in-sync replica remains. Reads have matching read concerns: local, majority, and linearizable, and a read that cannot satisfy its concern fails rather than returning a weaker result.

Conflict resolution

Normal writes are serialised through one primary per replication group. When a receiver applies a batch and finds the target row already present, it passes the local and incoming versions to the configured resolver:

StrategyClassBehaviour
Last-Writer-WinsLWWResolverThe resolver selects the change with the higher HLC timestamp, breaking ties by node ID.
Field-Level MergeFieldMergeResolverThe resolver merges non-overlapping columns and uses per-column HLC metadata for overlapping ones.
Primary WinsPrimaryWinsResolverThe resolver selects the version authored by a configured primary node ID.

Custom resolvers define a single resolve(ctx) method, and resolvers can be set per table.

Coordinator-backed failover

Coordinator mode stores primary authority, node sessions, replication-group state, and the in-sync set in a cluster coordinator, so authority survives a primary loss. The package includes an etcd adapter. Build the coordinator from the etcd hosts and node credentials, then reference it from a coordinator config block that names the cluster, the replication group, and the voting nodes.

import { ChangeTracker, Sirannon } from '@delali/sirannon-db'
import { betterSqlite3 } from '@delali/sirannon-db/driver/better-sqlite3'
import { ReplicationEngine, PrimaryReplicaTopology } from '@delali/sirannon-db/replication'
import { GrpcReplicationTransport } from '@delali/sirannon-db/transport/grpc'
import { createEtcdCoordinator } from '@delali/sirannon-db/replication/coordinator/etcd'
import { readFileSync } from 'node:fs'
 
const dbPath = './data/orders.db'
const driver = betterSqlite3()
const sirannon = new Sirannon({ driver })
const db = await sirannon.open('orders', dbPath)
const writerConn = await driver.open(dbPath)
const tracker = new ChangeTracker({ replication: true })
 
const transport = new GrpcReplicationTransport({
  host: '0.0.0.0',
  port: 4200,
  tlsCert: './certs/primary.crt',
  tlsKey: './certs/primary.key',
  tlsCaCert: './certs/ca.crt',
})
 
const coordinator = createEtcdCoordinator({ 
  hosts: ['https://etcd-1.internal:2379', 'https://etcd-2.internal:2379'],
  keyPrefix: '/sirannon/orders',
  credentials: {
    rootCertificate: readFileSync('./certs/etcd-ca.crt'),
    privateKey: readFileSync('./certs/orders-node.key'),
    certChain: readFileSync('./certs/orders-node.crt'),
  },
})
 
const engine = new ReplicationEngine(db, writerConn, {
  nodeId: 'primary-us-east-1',
  topology: new PrimaryReplicaTopology('primary'),
  transport,
  coordinator: { 
    clusterId: 'commerce-production',
    groupId: 'orders',
    endpoint: 'https://primary-us-east-1.internal/db/orders',
    coordinator,
    votingDataBearingNodeIds: ['primary-us-east-1', 'replica-eu-west-1', 'replica-ap-south-1'],
    controller: true,
  },
  snapshotConnectionFactory: () => driver.open(dbPath, { readonly: true }),
  changeTracker: tracker,
})
 
await engine.start()
 
await engine.execute('CREATE TABLE IF NOT EXISTS accounts (id INTEGER PRIMARY KEY, holder TEXT NOT NULL, balance INTEGER NOT NULL DEFAULT 0)')
await engine.execute('CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, total INTEGER NOT NULL)')
 
await engine.executeBatch('INSERT INTO accounts (id, holder, balance) VALUES (?, ?, ?)', [
  [1, 'Emma Wright', 5000],
  [2, 'Kenji Tanaka', 2000],
])

A primary may accept writes only while it can prove current authority through terms and leases. Replicas reject stale batches and stale forwarded writes, and only an in-sync replica can be promoted. Automatic write failover needs at least three voting data-bearing nodes; with fewer, a survivor cannot prove majority authority and writes stay unavailable rather than risking a split brain.

A returning former primary with local-only writes is quarantined for manual review, and Sirannon never merges its writes automatically. An operator rebuilds or restores the node before it rejoins.

Schema changes

Replicated DDL passes a safety allowlist: CREATE TABLE, ALTER TABLE ... ADD COLUMN, DROP TABLE, CREATE INDEX, and DROP INDEX. The receiver rejects multiple statements, AS SELECT, extension loading, ATTACH, and dangerous file functions.

The full configuration reference, including batch sizes, sync timeouts, flow control, and the coordinator options, is in the replication section of the README. The distributed entitlements example runs a three-node coordinator-backed cluster with etcd, mutual TLS, and failure injection through Toxiproxy.