Coordinator-backed failover is stable. In coordinator mode, Sirannon uses etcd to establish which node may accept writes. A controller can promote an eligible replica when the current primary fails.
Each group replicates one database. Keep at least three voting data-bearing nodes when the group must retain a majority after losing one node. A group holds at most seven voters, counting the pending ones, so the coordinator refuses to create a larger group or to add an eighth voter. A live session alone does not establish that a node has the data required for promotion.
Prepare the group and its credentials
Install the optional peers for etcd, gRPC, and the server alongside the database driver:
pnpm add -E @delali/sirannon-db better-sqlite3 etcd3 @grpc/grpc-js @bufbuild/protobuf grpc-health-check
pnpm add -E 'uWebSockets.js@github:uNetworking/uWebSockets.js#v20.69.0'Provide an HTTPS etcd cluster and its client credentials. The etcd identity needs access to the configured key prefix. These credentials are separate from the certificates used between replication peers.
Create a replication certificate for each node, following the mutual TLS setup. Include replica-ap-south-1 when issuing the third certificate. Each certificate's common name must match the node's stable identifier.
The example below places all three processes on one host with different ports and files. In a deployment across hosts, replace the loopback bind addresses, advertised addresses, and certificate alternative names together.
Start each node with a stable identifier
Save this complete file as coordinator-node.ts. Each process uses the same file with a different SIRANNON_NODE_ID. Set SIRANNON_OPERATOR_TOKEN to a shared operator credential and SIRANNON_ETCD_URLS to a comma-separated list of HTTPS etcd endpoints. Set the three SIRANNON_ETCD_* certificate paths for the etcd client.
import { timingSafeEqual } from 'node:crypto'
import { mkdir, readFile } from 'node:fs/promises'
import { type OperationRegistry, type RequestContext, type SQLiteConnection, RequestDeniedError, Sirannon, ChangeTracker } from '@delali/sirannon-db'
import { betterSqlite3 } from '@delali/sirannon-db/driver/better-sqlite3'
import {
PrimaryReplicaTopology,
ReplicationEngine,
toClusterStatusInfo,
toReplicationStatusInfo,
} from '@delali/sirannon-db/replication'
import { createEtcdCoordinator } from '@delali/sirannon-db/replication/coordinator/etcd'
import { createServer } from '@delali/sirannon-db/server'
import { GrpcReplicationTransport } from '@delali/sirannon-db/transport/grpc'
const required = (key: string): string => {
const value = process.env[key]
if (!value) throw new Error(`Set ${key}`)
return value
}
const nodeId = required('SIRANNON_NODE_ID')
const isFirst = nodeId === 'primary-us-east-1'
const isSecond = nodeId === 'replica-eu-west-1'
if (!isFirst && !isSecond && nodeId !== 'replica-ap-south-1') throw new Error('Unknown node identifier')
const peerPort = isFirst ? 4200 : isSecond ? 4201 : 4202
const httpPort = isFirst ? 9876 : isSecond ? 9877 : 9878
const expectedToken = Buffer.from(`Bearer ${required('SIRANNON_OPERATOR_TOKEN')}`)
const isOperator = (context: RequestContext): boolean => {
const supplied = Buffer.from(context.headers.authorization ?? '')
return supplied.length === expectedToken.length && timingSafeEqual(supplied, expectedToken)
}
const authenticate = (context: RequestContext): { role: string } => {
if (!isOperator(context)) throw new RequestDeniedError(401, 'UNAUTHORIZED', 'Operator credential required')
return { role: 'operator' }
}
const coordinator = createEtcdCoordinator({
hosts: required('SIRANNON_ETCD_URLS').split(',').map((url: string): string => url.trim()),
keyPrefix: '/sirannon/orders',
credentials: {
rootCertificate: await readFile(required('SIRANNON_ETCD_CA')),
privateKey: await readFile(required('SIRANNON_ETCD_KEY')),
certChain: await readFile(required('SIRANNON_ETCD_CERT')),
},
defaultCallTimeoutMs: 5000,
})
await mkdir('./data', { recursive: true })
const path = `./data/${nodeId}.db`
const driver = betterSqlite3()
const writer = await driver.open(path)
await writer.exec('CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, total INTEGER NOT NULL)')
const tracker = new ChangeTracker()
await tracker.watch(writer, 'orders')
const sirannon = new Sirannon({ driver })
const db = await sirannon.open('orders', path)
const engine = new ReplicationEngine(db, writer, {
nodeId,
topology: new PrimaryReplicaTopology(isFirst ? 'primary' : 'replica'),
transport: new GrpcReplicationTransport({
host: '127.0.0.1',
port: peerPort,
tlsCert: `./certs/${nodeId}.crt`,
tlsKey: `./certs/${nodeId}.key`,
tlsCaCert: './certs/ca.crt',
}),
changeTracker: tracker,
snapshotConnectionFactory: (): Promise<SQLiteConnection> => driver.open(path, { readonly: true }),
writeForwarding: true,
coordinator: {
clusterId: 'commerce',
groupId: 'orders',
endpoint: `http://127.0.0.1:${httpPort}`,
replicationEndpoint: `localhost:${peerPort}`,
votingDataBearingNodeIds: ['primary-us-east-1', 'replica-eu-west-1', 'replica-ap-south-1'],
coordinator,
controller: true,
},
})
const operations: OperationRegistry = {
orders: {
reads: {
listOrders: {
args: [],
columns: ['id', 'total'],
statement: (): { sql: string } => ({ sql: 'SELECT id, total FROM orders ORDER BY id' }),
},
},
},
}
const executionTarget = (id: string): ReplicationEngine | null => id === 'orders' ? engine : null
const clusterStatus = (id: string): ReturnType<typeof toClusterStatusInfo> | null =>
id === 'orders' ? toClusterStatusInfo(engine.status(), { databaseId: id }) : null
const replicationStatus = (): ReturnType<typeof toReplicationStatusInfo> => toReplicationStatusInfo(engine.status())
const server = createServer(sirannon, {
host: '127.0.0.1',
port: httpPort,
authenticate,
authorizeClusterStatus: isOperator,
operations,
resolveExecutionTarget: executionTarget,
getClusterStatus: clusterStatus,
getReplicationStatus: replicationStatus,
})
let stopping: Promise<void> | undefined
const stop = (): Promise<void> => {
stopping ??= server.close().finally((): Promise<void> =>
engine.stop().finally((): Promise<void> =>
writer.close().finally((): Promise<void> =>
sirannon.shutdown().finally((): Promise<void> => coordinator.close()))))
return stopping
}
const onSignal = (): void => {
stop().catch((error: unknown): void => {
console.error(error)
process.exitCode = 1
})
}
process.once('SIGINT', onSignal)
process.once('SIGTERM', onSignal)
try {
await engine.start()
await server.listen()
console.log(engine.status())
} catch (error) {
await stop()
throw error
}Start etcd first, then the process configured as primary-us-east-1, then the replicas. Query readiness and cluster status to confirm admission before relying on majority reads or writes. This example requires a live etcd deployment and certificates, so it has no fixed result panel.
The example server binds to loopback and registers only listOrders. Add application writes through registered operations, and place TLS termination in front of the HTTP server before exposing it beyond the host.
Discover addresses through sessions
Every node publishes two addresses. Clients use endpoint, while peer transports use replicationEndpoint. Sirannon reads each address from that member's live coordinator session.
Coordinator mode refuses transportConfig.endpoints and a configured client-address map passed to toClusterStatusInfo, with TOPOLOGY_ERROR. Static mode continues to accept configured addresses.
A restarted process can replace its earlier session under the same nodeId; it need not wait for the earlier lease to expire. Ensure that only the process using that durable copy uses its identifier. A replacement session is not a mechanism for sharing one identifier between independent nodes.
Count only copies that can prove their data
On restart, a replica leaves the in-sync set before registering its session. A node with no completed copy marks itself repairing and copies from the primary. A restarted primary retains its duty only when its file records the current term and reaches the group's durability point.
A restarted primary also accepts no write until it proves that it holds every acknowledged write, and it then stays primary in a new term. While another voter remains in the in-sync set, one report from that voter at or below the primary's newest write settles the proof. When no other voter remains in the set, as when each of them restarts, the primary needs reports from enough of the remaining voters to include one that holds every acknowledged write. When the restarted primary is the last copy in the set and another voter reports a higher position, the primary marks itself repairing, and the group has no safe primary until an operator writes the group's state.
A replica's new connection resets the acknowledgements counted for that replica. The primary admits a rebuilt or returning copy only after it proves the required position in the current history. A process being reachable therefore does not make it safe to promote.
The current primary records the start of its term before acknowledging writes or admitting replicas. A stale primary cannot advance the group's durability point or membership under another term. If no replica qualifies for promotion, the group refuses writes with NO_SAFE_PRIMARY.
When the primary confirms a majority write
A coordinator-mode write that states no concern uses majority. The primary confirms such a write once every copy in the in-sync set holds it, counting its own commit, and only while the set holds most of the voters. In a group at feature level 2, the primary records nothing in etcd for each write, so a write adds no coordinator transaction.
Because the primary waits for every copy in the set, a replica that stalls delays every majority write until the primary removes it. The primary removes an in-sync replica that leaves a change unacknowledged for longer than coordinator.maxReplicaLagMs, which is 1,000 milliseconds by default. It removes only as many voters as leave a majority in the set, starting with the one furthest behind, and the coordinator refuses any removal below that majority. Keep maxReplicaLagMs well below the default write-concern timeout of 5,000 milliseconds, so that a stalled replica leaves the set before the writes that wait on it time out.
When two copies of a three-voter group stall or drain, majority writes fail until one of them returns, because the set then holds only one of the three voters. Only the primary admits a copy back to the set, once the copy holds both the durability point and the highest write that the primary acknowledges. A replica outside the set reports its position every second, so that the primary can admit it while no writes come in.
Reconcile history after failover
A returning replica compares its recorded history with the current primary. Where they share a usable history point, Sirannon can undo later local history and resume from that point. A replica that cannot safely undo its history needs a full copy.
A former primary that returns with writes that only it holds undoes those writes and rejoins the group without an operator. In the same transaction, it keeps a copy of each undone change in the _sirannon_rolled_back_changes table inside the database file, so in an encrypted database the copies stay encrypted with the rest of the data. Sirannon never confirms such a write at majority, because the write never reaches a majority of the voters. A read at local concern from the former primary can return such a write until the former primary undoes it.
rolledBackChanges(engine) from @delali/sirannon-db/replication returns the kept changes, oldest first, so that you can review them and apply again the ones that still matter. clearRolledBackChanges(engine) deletes them once you review them. The node keeps every row until you clear the table, so the table grows with each failover that undoes writes.
Each RolledBackChange holds the table, the operation, the row identifier, the row images before and after the change, the writing node, the transaction, its HLC, and the term in which the node undid it. comparedWithPrimary is true when the node compared its history with the primary's and found the change missing there. It is false when the primary holds no history to compare against, so the primary may hold the change as well. Before a node rebuilds through a full copy, it keeps in the same table every change that the primary may lack.
An operator can mark a node faulted, which keeps the node out of the read, write, and promotion paths. Once the operator clears the fault and marks the node repairing, the node must complete a full copy before admission.
The failover contract for 0.4.1 specifies history reconciliation, the feature levels below, and the compatibility rules.
Upgrade a group from 0.4.0
Upgrade a group from 0.4.0 one node at a time. Restart each node in place under the same nodeId, and wait for it to return to the in-sync set before you move to the next one. Under a steady stream of majority writes, a 0.4.0 primary can keep only itself in the in-sync set. While that primary restarts, the controller therefore finds no replica to promote, and writes fail with NO_SAFE_PRIMARY until the primary starts again on 0.4.1 with its rows in place.
A group's feature level records which rules its copies follow. Each node registers the highest level that it supports in its session, which is 2 for 0.4.1. A group that a 0.4.0 node creates holds level 1, at which the primary records the durability point of every majority write in etcd. A group that a 0.4.1 node creates starts at level 2, at which the primary records the durability point only when its term starts.
Once every node of an upgraded group holds a live session on 0.4.1, raise the group to level 2 with raiseFeatureLevel({ clusterId, groupId, featureLevel: 2 }) on the coordinator. The coordinator refuses with a RangeError while any member lacks a live session that supports the level, and it refuses a level lower than the group's. Start no 0.4.0 node once the group holds level 2, because a 0.4.0 node drops the feature level when it rewrites the group's record. A node refuses to start in a group whose level exceeds the highest one that it supports, with PROTOCOL_VERSION_MISMATCH.
Move a copy to a new disk
Use updateNodeMaintenance to mark the node draining. Wait for another node to become primary if necessary and for the drained node to leave the in-sync set. Stop that node, then start its replacement under the same identifier.
An empty replacement marks itself repairing and copies from the primary. Clear the drain when the replacement is ready to rejoin. The primary admits it only after the completed sync establishes that it reaches the durability point.
Add or remove a voter
addVoter and removeVoter take expectedVoterNodeIds, the voter list that the operator reads before requesting the change. If another change commits first, the operation returns changed: false; reread the group before retrying.
An added node becomes a pending voter and counts towards no majority. Once its full copy completes, admission moves it into the voters and in-sync set together. While a pending voter exists, the coordinator accepts only its removal as another voter change.
The coordinator refuses to remove the current primary or to leave fewer in-sync voters than the remaining majority requires. Removing a pending voter changes neither count, so an operator can remove it even when the group lacks a majority.
A removed node refuses reads and writes with TOPOLOGY_ERROR, and the primary sends it no further changes. The primary counts a waiting write against the voters in the newest group state that it holds when it confirms the write, so a membership change before that moment applies to the write.
Monitor authority and stop in order
One active controller establishes failover decisions under its controller lease. Standby controllers watch that lease where the coordinator supports the watch; otherwise they bid at the configured tick interval. A transport disconnect alone does not establish that the primary's coordinator lease is expired.
GET /health/ready reports the local node's replication health. GET /db/orders/cluster reports client routing information, including available read concerns, through the authenticated operator route in this example. A removed node reports not-member health.
Stop the server before stopping the engine, closing its separate writer, and shutting down the registry. Configure coordinator call deadlines, as in the example, so that an unavailable dependency cannot leave a coordinator call waiting indefinitely.
Use coordinator-backed backup membership to exclude draining, repairing, and faulted copies from backup selection. Stop an engine before an encryption change or restore that replaces its database file.