> Complete content index: https://sirannon.sondelali.com/llms.txt

# Backups in a replication group

> Give every node of a replication group the same backup settings and let one of them take the backups, with a preference you set and a skip report you can alert on.

URL: https://sirannon.sondelali.com/docs/backups-in-a-cluster
Section: Backups
Version: 0.3 (@delali/sirannon-db@0.3.0, latest)

Every node of a [replication group](/docs/distributed-replication) holds the same data, so backing up on all of them writes the same bytes several times over, and your destination stores each of those copies. Naming one node in configuration means editing that node's settings whenever a failover moves the work, and a group that fails over before anyone edits them backs up nowhere until an operator notices.

Sirannon settles the question at each turn. Every node takes the same `backups` options, and each turn asks the group who should take the backups. The node that finds its own identifier in the answer copies, while the others stand down, so a failover changes which node copies and leaves every node's configuration alone.

This page builds one file, `group-backup.ts`, step by step. You will give the cycle a group to ask, watch a node stand down, and then take the other node out of the group and watch the first one pick the work up. The [chains page](/docs/backup-chains) covers the cycle itself, which is unchanged here.

```bash
rm -rf data && mkdir -p data
```

## What the cycle asks the group

`replicationGroup` takes a **backup group source**, which is an object stating the `nodeId` this node is known by and offering a `readMembership` method that reports who is in the group at this moment.

```ts
interface BackupGroupSource {
  readonly nodeId: string
  readMembership(): Promise<{ primaryNodeId: string | null; nodeIds: string[] }>
}
```

`preferredNode` says which of those nodes you want the work on, and Sirannon resolves it the same way on every node, because every node computes the answer from the same membership and Sirannon sorts the eligible identifiers first.

| `preferredNode` | Resolves to |
| --- | --- |
| `'replica'` (the default) | The lowest eligible identifier other than the primary, and the primary where the group offers no other eligible node. |
| `'primary'` | The node the group names primary where that node is eligible, and no node at all otherwise. |
| `{ nodeId: 'frankfurt-1' }` | That identifier. Sirannon reads no membership for this form, so the node matching it needs no coordinator to answer. |

The default puts the work on a replica, so the node serving writes keeps serving them while another node does the copying. A group with only its primary left falls back to that primary, which is what stops a group backing up nowhere.

Leave `replicationGroup` out on a single-node deployment. Every turn then falls to the only node there is.

## Watch a node stand down

The source below is written by hand, so this page runs as one file. It reports a group of two nodes with `frankfurt-1` as the primary, and it runs on `frankfurt-1`, so the default preference puts the backups on `singapore-2` and this node stands down.

`onSkip` reports every turn the node stands down from. Collect the skips and read the last one.

```ts title="group-backup.ts"
import {
  type BackupDestination,
  type BackupGroupMembership,
  type BackupGroupSource,
  type BackupPiece,
  type BackupSkip,
  Sirannon,
} from '@delali/sirannon-db'

const stored = new Map<string, Uint8Array>()
const pieceKey = (name: string, index: number): string => `${name}#${index}`

const memory: BackupDestination = {
  async writePiece(name, index, bytes) {
    stored.set(pieceKey(name, index), Uint8Array.from(bytes))
  },
  async writePieceIfAbsent(name, index, bytes) {
    if (stored.has(pieceKey(name, index))) return false
    stored.set(pieceKey(name, index), Uint8Array.from(bytes))
    return true
  },
  async readPiece(name, index) {
    const bytes = stored.get(pieceKey(name, index))
    if (!bytes) throw new Error(`No piece ${index} of '${name}'`)
    return bytes
  },
  async listPieces(name) {
    const pieces: BackupPiece[] = []
    for (const [key, bytes] of stored) {
      if (key.startsWith(`${name}#`)) {
        pieces.push({ index: Number(key.slice(name.length + 1)), byteLength: bytes.byteLength })
      }
    }
    return pieces
  },
}

let membership: BackupGroupMembership = { primaryNodeId: 'frankfurt-1', nodeIds: ['frankfurt-1', 'singapore-2'] }

const group: BackupGroupSource = {
  nodeId: 'frankfurt-1',
  readMembership: async () => membership,
}

const skips: BackupSkip[] = []

const sirannon = new Sirannon({ driver: betterSqlite3() })

const db = await sirannon.open('orders', './data/orders.db', {
  backups: {
    destination: memory,
    intervalMs: 0,
    replicationGroup: group,
    preferredNode: 'replica',
    onSkip: (skip) => skips.push(skip),
    onError: (err) => console.error('Backup cycle failed:', err.message),
  },
})

await db.execute('CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, customer TEXT NOT NULL, total REAL NOT NULL)')
await db.execute('INSERT INTO orders (customer, total) VALUES (?, ?)', ['Amara Okonkwo', 128.5])

const whileTheReplicaIsUp = await db.captureBackupChanges()

console.log(
  JSON.stringify(
    {
      whileTheReplicaIsUp: {
        wroteAnything: whileTheReplicaIsUp !== undefined,
        lastSkip: {
          reason: skips[skips.length - 1]?.reason,
          nodeId: skips[skips.length - 1]?.nodeId,
          preferredNodeId: skips[skips.length - 1]?.preferredNodeId,
        },
      },
    },
    null,
    2,
  ),
)

await sirannon.shutdown()
```

```json result
{
  "whileTheReplicaIsUp": {
    "wroteAnything": false,
    "lastSkip": {
      "reason": "not-preferred",
      "nodeId": "frankfurt-1",
      "preferredNodeId": "singapore-2"
    }
  }
}
```

`captureBackupChanges` returns `undefined`, because this turn wrote nothing at all. The skip names both nodes, so a log line built from it says who stood down and who was expected to do the work.

A node that stands down takes three steps before it reports the skip. It sends any capture it had staged, so bytes it already produced still reach the destination. It deletes its own cycle state, so it holds no chain. It checkpoints the log, so the disk under it stops growing. Where the destination refuses that staged capture, the node keeps the chain, the capture, and the log, and stands down on a later turn.

## Watch the work move on a failover

With `singapore-2` out of the membership, the same node resolves to itself on the next turn.

```ts title="group-backup.ts"
import {
  type BackupDestination,
  type BackupGroupMembership,
  type BackupGroupSource,
  type BackupPiece,
  type BackupSkip,
  Sirannon,
} from '@delali/sirannon-db'

const stored = new Map<string, Uint8Array>()
const pieceKey = (name: string, index: number): string => `${name}#${index}`

const memory: BackupDestination = {
  async writePiece(name, index, bytes) {
    stored.set(pieceKey(name, index), Uint8Array.from(bytes))
  },
  async writePieceIfAbsent(name, index, bytes) {
    if (stored.has(pieceKey(name, index))) return false
    stored.set(pieceKey(name, index), Uint8Array.from(bytes))
    return true
  },
  async readPiece(name, index) {
    const bytes = stored.get(pieceKey(name, index))
    if (!bytes) throw new Error(`No piece ${index} of '${name}'`)
    return bytes
  },
  async listPieces(name) {
    const pieces: BackupPiece[] = []
    for (const [key, bytes] of stored) {
      if (key.startsWith(`${name}#`)) {
        pieces.push({ index: Number(key.slice(name.length + 1)), byteLength: bytes.byteLength })
      }
    }
    return pieces
  },
}

let membership: BackupGroupMembership = { primaryNodeId: 'frankfurt-1', nodeIds: ['frankfurt-1', 'singapore-2'] }

const group: BackupGroupSource = {
  nodeId: 'frankfurt-1',
  readMembership: async () => membership,
}

const skips: BackupSkip[] = []

const sirannon = new Sirannon({ driver: betterSqlite3() })

const db = await sirannon.open('orders', './data/orders.db', {
  backups: {
    destination: memory,
    intervalMs: 0,
    replicationGroup: group,
    preferredNode: 'replica',
    onSkip: (skip) => skips.push(skip),
    onError: (err) => console.error('Backup cycle failed:', err.message),
  },
})

await db.execute('CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, customer TEXT NOT NULL, total REAL NOT NULL)')
await db.execute('INSERT INTO orders (customer, total) VALUES (?, ?)', ['Amara Okonkwo', 128.5])

const whileTheReplicaIsUp = await db.captureBackupChanges()
// [!code ++:6]

membership = { primaryNodeId: 'frankfurt-1', nodeIds: ['frankfurt-1'] }

const onceTheReplicaHasGone = await db.captureBackupChanges()

const chains = await db.backupChain()

console.log(
  JSON.stringify(
    {
      whileTheReplicaIsUp: {
        wroteAnything: whileTheReplicaIsUp !== undefined,
        lastSkip: {
          reason: skips[skips.length - 1]?.reason,
          nodeId: skips[skips.length - 1]?.nodeId,
          preferredNodeId: skips[skips.length - 1]?.preferredNodeId,
        },
      },
      // [!code ++:7]
      onceTheReplicaHasGone: {
        kind: onceTheReplicaHasGone?.kind,
        route: onceTheReplicaHasGone?.route,
        chainsAtTheDestination: chains.length,
        chainHoldsTheFullCopy: chains[0]?.base?.runId === onceTheReplicaHasGone?.runId,
        skipsSoFar: skips.length,
      },
    },
    null,
    2,
  ),
)

await sirannon.shutdown()
```

```json result
{
  "whileTheReplicaIsUp": {
    "wroteAnything": false,
    "lastSkip": {
      "reason": "not-preferred",
      "nodeId": "frankfurt-1",
      "preferredNodeId": "singapore-2"
    }
  },
  "onceTheReplicaHasGone": {
    "kind": "full",
    "route": "staged",
    "chainsAtTheDestination": 1,
    "chainHoldsTheFullCopy": true,
    "skipsSoFar": 2
  }
}
```

The turn after the failover reports `kind: 'full'`, because a node that stood down holds no cycle state and a node holding none starts a fresh chain with a full copy. A chain therefore continues on the node that started it and on no other, and moving the backups between nodes takes one full copy.

`skipsSoFar` reads two, because the turn the cycle takes as it starts also stood down. That first turn is why a node joining a group already backing up elsewhere writes nothing at the destination.

## Report every skip, and alert on the log

`onSkip` fires on every turn the cycle passes over, and `reason` names which of three conditions held.

| `reason` | What happened |
| --- | --- |
| `'not-preferred'` | Another node of the group takes these backups. |
| `'group-unavailable'` | `readMembership` failed, so this node could not tell whose turn it was. It captures nothing and checkpoints nothing. |
| `'previous-run-active'` | A scheduled turn fell due while the turn before it was still running. |

Every skip also states `uncapturedLogBytes`, which is the size of the write-ahead log at the moment the node passed the turn over. That figure is worth a metric, because a node holding a log that keeps growing is a node whose group backs up nowhere, and you hear about it long before `maxUncapturedLogBytes` ends the chain.

A `'group-unavailable'` skip that repeats points at a coordinator outage. The node keeps its chain and its log through it, so the group continues where it left off once the coordinator answers again.

## The destination has to claim a place

Two nodes of a group can start a chain at the same moment during a failover, and each would append its chain to the same list of chains. Where the destination can claim a name, Sirannon claims each place through `writePieceIfAbsent` and moves to the next index wherever the claim reports false, so both chains reach the list.

With a destination that offers no `writePieceIfAbsent`, Sirannon writes the record and reads it back, which loses a chain where the other node's write falls between those two calls. Sirannon therefore reports that arrangement as the cycle starts. A cycle whose options name a `replicationGroup` and whose destination offers no `writePieceIfAbsent` raises `BACKUP_DESTINATION_ERROR` through `onError`, and the message says to add the function or to give each node a `chainName` of its own.

## Read the membership from your cluster coordinator

A group that already uses [coordinator-backed failover](/docs/distributed-replication#coordinator-backed-failover) keeps its membership in the coordinator, and `coordinatorBackupGroup` reads it from there. It offers the nodes the group counts as in sync, less any node being drained, rebuilt, or held out as faulted, so a node that falls behind never takes the backups.

Build the coordinator first, pass the group source to `sirannon.open`, and build the replication engine afterwards with the same `nodeId`. The destination here is the same one this page has used throughout, and a real deployment puts object storage behind those three methods.

```ts title="coordinator-backup.ts"

const stored = new Map<string, Uint8Array>()
const pieceKey = (name: string, index: number): string => `${name}#${index}`

const destination: BackupDestination = {
  async writePiece(name, index, bytes) {
    stored.set(pieceKey(name, index), Uint8Array.from(bytes))
  },
  async writePieceIfAbsent(name, index, bytes) {
    if (stored.has(pieceKey(name, index))) return false
    stored.set(pieceKey(name, index), Uint8Array.from(bytes))
    return true
  },
  async readPiece(name, index) {
    const bytes = stored.get(pieceKey(name, index))
    if (!bytes) throw new Error(`No piece ${index} of '${name}'`)
    return bytes
  },
  async listPieces(name) {
    const pieces: BackupPiece[] = []
    for (const [key, bytes] of stored) {
      if (key.startsWith(`${name}#`)) {
        pieces.push({ index: Number(key.slice(name.length + 1)), byteLength: bytes.byteLength })
      }
    }
    return pieces
  },
}

const coordinator = createEtcdCoordinator({
  hosts: ['https://etcd-1.internal:2379', 'https://etcd-2.internal:2379'],
  keyPrefix: '/sirannon/orders',
  credentials: {
    rootCertificate: readFileSync('./certs/etcd-ca.crt'),
    privateKey: readFileSync('./certs/orders-node.key'),
    certChain: readFileSync('./certs/orders-node.crt'),
  },
})

const sirannon = new Sirannon({ driver: betterSqlite3() })

const db = await sirannon.open('orders', './data/orders.db', {
  backups: {
    destination,
    replicationGroup: coordinatorBackupGroup({
      coordinator,
      clusterId: 'commerce-production',
      groupId: 'orders',
      nodeId: 'primary-us-east-1',
    }),
    onSkip: (skip) => console.log('backup.uncaptured_log_bytes', skip.uncapturedLogBytes ?? 0),
    onError: (err) => console.error('Backup cycle failed:', err.message),
  },
})

console.log(db.backupCapabilities().fullCopy)
```

That file needs a live etcd cluster and the certificates the [replication guide](/docs/distributed-replication#coordinator-backed-failover) creates, so this page shows no captured output for it. The `nodeId` you give it has to match the one the replication engine runs under, because that identifier is what the coordinator records and what the membership answers with.

Every node of the group runs the same file. One of them finds itself named and copies, while the rest report a skip and checkpoint their logs.