Backup destinations

Sirannon writes a backup through a destination you supply. A destination is an object with three methods that move bytes, and Sirannon calls them with fixed-size pieces of the copy. Object storage, another machine, and a directory on the same disk all fit behind those three methods, which keeps your credentials and your retry policy inside your own code.

This page builds one file, copy-to-storage.ts, step by step. You write a destination backed by a folder, copy a database into it, watch the copy move, and read the pieces back into a database you can query. The backups page covers the copy to a local file that this one builds on, and the chains page covers repeating the copy so that each run sends only what changed.

rm -rf data && mkdir -p data

What a destination has to do

A destination provides three operations. It stores one piece, it gives one stored piece back, and it lists the pieces it holds under a given name. Sirannon numbers the pieces from zero, and every piece holds pieceBytes bytes except the last.

interface BackupDestination {
  writePiece(name: string, index: number, bytes: Uint8Array): Promise<void>
  readPiece(name: string, index: number): Promise<Uint8Array>
  listPieces(name: string): Promise<{ index: number; byteLength: number }[]>
  writePieceIfAbsent?(name: string, index: number, bytes: Uint8Array): Promise<boolean>
}

Four rules follow from how SQLite writes a database, and a destination that breaks one of them produces a backup no restore can use.

Accept the pieces in any order, because SQLite writes page one last.
Let a second write to the same name and index replace the piece already stored, because a run resumed after an interruption repeats its last write.
Return from readPiece whatever the most recent writePiece to that name and index stored, because Sirannon reads a record back to confirm that no other node replaced it.
Hold more than one name, because a chain stores its full copy, each change piece, and its own list under a name of its own.

writePieceIfAbsent is optional. It stores a piece only where that name and index hold none and reports whether this call is the one that stored it, which is how two nodes of a replication group claim separate places in the same list. The cluster page covers when you need it.

Copy a database into your own storage

The destination below writes each piece as its own file in ./data/pieces, named after the backup and its index. A destination backed by an object-storage client implements the same three methods with the same signatures. db.backupTo then copies the database into it.

copy-to-storage.ts

import { mkdir, readdir, readFile, stat, writeFile } from 'node:fs/promises'
import { join } from 'node:path'
import { type BackupDestination, type BackupPiece, Sirannon } from '@delali/sirannon-db'
import { betterSqlite3 } from '@delali/sirannon-db/driver/better-sqlite3'
 
const pieceDir = './data/pieces'
const pieceFile = (name: string, index: number): string => join(pieceDir, `${name}.${String(index).padStart(6, '0')}`)
 
const folder: BackupDestination = {
  async writePiece(name, index, bytes) {
    await mkdir(pieceDir, { recursive: true })
    await writeFile(pieceFile(name, index), bytes)
  },
  async readPiece(name, index) {
    return new Uint8Array(await readFile(pieceFile(name, index)))
  },
  async listPieces(name) {
    const entries = await readdir(pieceDir).catch(() => [])
    const pieces: BackupPiece[] = []
    for (const entry of entries) {
      if (!entry.startsWith(`${name}.`)) continue
      const { size } = await stat(join(pieceDir, entry))
      pieces.push({ index: Number(entry.slice(name.length + 1)), byteLength: size })
    }
    return pieces
  },
}
 
const driver = betterSqlite3()
const sirannon = new Sirannon({ driver })
 
const db = await sirannon.open('orders', './data/orders.db')
 
await db.execute('CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, customer TEXT NOT NULL, total REAL NOT NULL)')
 
const ORDER_COUNT = 5000
await db.executeBatch(
  'INSERT INTO orders (customer, total) VALUES (?, ?)',
  Array.from({ length: ORDER_COUNT }, (_, index) => [`Customer ${index + 1}`, (index % 400) + 0.5]),
)
 
const run = await db.backupTo({
  destination: folder,
  name: 'orders-full.db',
  pieceBytes: 65_536,
})
 
await sirannon.shutdown()
 
console.log(
  JSON.stringify(
    {
      copy: {
        kind: run.kind,
        destinationName: run.destinationName,
        pageCount: run.pageCount,
        pageSize: run.pageSize,
        bytesWritten: run.bytesWritten,
        pieceCount: run.pieceCount,
        pieceBytes: run.pieceBytes,
        restarts: run.restarts,
      },
    },
    null,
    2,
  ),
)

Five thousand rows come to 39 pages of 4,096 bytes, which is 159,744 bytes, and a piece size of 64 KiB divides that into three pieces. A larger ORDER_COUNT raises the page count, the byte total, and the piece count together, while a different pieceBytes moves the piece count alone.

The report states more fields than the panel prints. runId names the run; startedAt, finishedAt, durationMs, copyMs, and transferMs time it; fingerprint holds the SHA-256 of what it wrote; route names which of the two routes the run took; and chainId names the chain this copy begins. A copy taken this way begins a chain of one, and the chains page covers how that chain grows.

destinationName matters when you come to read the backup back, because a run writes that one name and no other. The journal SQLite opens beside a copy, meanwhile, stays on local disk and never reaches your destination.

Watch a large copy as it moves

A full copy of a large database to remote storage can take long enough to need a progress display. onProgress reports once per copy step while SQLite moves pages, then once per piece while those pieces go out, and phase tells the two apart.

Keep the latest report of each phase and print both at the end.

copy-to-storage.ts

import { mkdir, readdir, readFile, stat, writeFile } from 'node:fs/promises'
import { join } from 'node:path'
import { type BackupDestination, type BackupPiece, Sirannon } from '@delali/sirannon-db'
import { type BackupDestination, type BackupPiece, type BackupProgress, Sirannon } from '@delali/sirannon-db'
import { betterSqlite3 } from '@delali/sirannon-db/driver/better-sqlite3'
 
const pieceDir = './data/pieces'
const pieceFile = (name: string, index: number): string => join(pieceDir, `${name}.${String(index).padStart(6, '0')}`)
 
const folder: BackupDestination = {
  async writePiece(name, index, bytes) {
    await mkdir(pieceDir, { recursive: true })
    await writeFile(pieceFile(name, index), bytes)
  },
  async readPiece(name, index) {
    return new Uint8Array(await readFile(pieceFile(name, index)))
  },
  async listPieces(name) {
    const entries = await readdir(pieceDir).catch(() => [])
    const pieces: BackupPiece[] = []
    for (const entry of entries) {
      if (!entry.startsWith(`${name}.`)) continue
      const { size } = await stat(join(pieceDir, entry))
      pieces.push({ index: Number(entry.slice(name.length + 1)), byteLength: size })
    }
    return pieces
  },
}
 
const driver = betterSqlite3()
const sirannon = new Sirannon({ driver })
 
const db = await sirannon.open('orders', './data/orders.db')
 
await db.execute('CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, customer TEXT NOT NULL, total REAL NOT NULL)')
 
const ORDER_COUNT = 5000
await db.executeBatch(
  'INSERT INTO orders (customer, total) VALUES (?, ?)',
  Array.from({ length: ORDER_COUNT }, (_, index) => [`Customer ${index + 1}`, (index % 400) + 0.5]),
)
 
const latest = new Map<BackupProgress['phase'], BackupProgress>()
 
const run = await db.backupTo({
  destination: folder,
  name: 'orders-full.db',
  pieceBytes: 65_536,
  onProgress: (progress) => latest.set(progress.phase, progress), 
})
 
await sirannon.shutdown()
 
console.log(
  JSON.stringify(
    {
      copy: {
        kind: run.kind,
        destinationName: run.destinationName,
        pageCount: run.pageCount,
        pageSize: run.pageSize,
        bytesWritten: run.bytesWritten,
        pieceCount: run.pieceCount,
        pieceBytes: run.pieceBytes,
        restarts: run.restarts,
      },
      progress: {
        totalPages: latest.get('copy')?.totalPages,
        remainingPages: latest.get('copy')?.remainingPages,
        piecesWritten: latest.get('transfer')?.piecesWritten,
        bytesWritten: latest.get('transfer')?.bytesWritten,
      },
    },
    null,
    2,
  ),
)

totalPages and remainingPages drive a copy bar, while piecesWritten against the eventual pieceCount drives a transfer bar. On the staged route every transfer report states remainingPages: 0, because the copy finishes before any piece goes out; on the streamed route the two phases interleave, so a transfer report states however many pages remain. How many reports fall between the first and the last depends on the route the bytes take and on how fast the disk answers, so read the counters for your display and treat the number of events as unspecified.

restarts rising during a copy is worth an alert, because another connection is writing to the source file and SQLite has sent the copy back to page one.

Turn the pieces back into a database

The pieces are a database file cut into numbered blocks, so putting them back together needs the index of each piece and nothing else. assembleFromDestination fetches them one at a time, writes each at index * pieceBytes, and checks the result against the report of the run that wrote them.

copy-to-storage.ts

import { mkdir, readdir, readFile, stat, writeFile } from 'node:fs/promises'
import { join } from 'node:path'
import { type BackupDestination, type BackupPiece, type BackupProgress, Sirannon } from '@delali/sirannon-db'
import { assembleFromDestination } from '@delali/sirannon-db/backup'
import { betterSqlite3 } from '@delali/sirannon-db/driver/better-sqlite3'
 
const pieceDir = './data/pieces'
const pieceFile = (name: string, index: number): string => join(pieceDir, `${name}.${String(index).padStart(6, '0')}`)
 
const folder: BackupDestination = {
  async writePiece(name, index, bytes) {
    await mkdir(pieceDir, { recursive: true })
    await writeFile(pieceFile(name, index), bytes)
  },
  async readPiece(name, index) {
    return new Uint8Array(await readFile(pieceFile(name, index)))
  },
  async listPieces(name) {
    const entries = await readdir(pieceDir).catch(() => [])
    const pieces: BackupPiece[] = []
    for (const entry of entries) {
      if (!entry.startsWith(`${name}.`)) continue
      const { size } = await stat(join(pieceDir, entry))
      pieces.push({ index: Number(entry.slice(name.length + 1)), byteLength: size })
    }
    return pieces
  },
}
 
const driver = betterSqlite3()
const sirannon = new Sirannon({ driver })
 
const db = await sirannon.open('orders', './data/orders.db')
 
await db.execute('CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, customer TEXT NOT NULL, total REAL NOT NULL)')
 
const ORDER_COUNT = 5000
await db.executeBatch(
  'INSERT INTO orders (customer, total) VALUES (?, ?)',
  Array.from({ length: ORDER_COUNT }, (_, index) => [`Customer ${index + 1}`, (index % 400) + 0.5]),
)
 
const latest = new Map<BackupProgress['phase'], BackupProgress>()
 
const run = await db.backupTo({
  destination: folder,
  name: 'orders-full.db',
  pieceBytes: 65_536,
  onProgress: (progress) => latest.set(progress.phase, progress),
})
 
await sirannon.shutdown()
 
const assembled = await assembleFromDestination(folder, run, './data/rebuilt.db')
 
const check = new Sirannon({ driver: betterSqlite3() })
const rebuilt = await check.open('rebuilt', './data/rebuilt.db')
const counted = await rebuilt.queryOne<{ orders: number }>('SELECT COUNT(*) AS orders FROM orders')
 
await check.shutdown()
 
console.log(
  JSON.stringify(
    {
      copy: {
        kind: run.kind,
        destinationName: run.destinationName,
        pageCount: run.pageCount,
        pageSize: run.pageSize,
        bytesWritten: run.bytesWritten,
        pieceCount: run.pieceCount,
        pieceBytes: run.pieceBytes,
        restarts: run.restarts,
      },
      progress: {
        totalPages: latest.get('copy')?.totalPages,
        remainingPages: latest.get('copy')?.remainingPages,
        piecesWritten: latest.get('transfer')?.piecesWritten,
        bytesWritten: latest.get('transfer')?.bytesWritten,
      },
      rebuilt: {
        pieceCount: assembled.pieceCount,
        bytesWritten: assembled.bytesWritten,
        fingerprintMatchesTheRun: assembled.fingerprint === run.fingerprint,
        orders: counted?.orders,
      },
    },
    null,
    2,
  ),
)

All 5,000 rows come back, and the digest of the assembled file equals the digest the run recorded. assembleFromDestination checks the listing before it opens the local file, so Sirannon refuses a destination that is missing a piece while the path you named still holds whatever it held. Once that file is open, Sirannon removes it after any failure, because a database missing its middle would otherwise stay on disk looking finished.

Four conditions fail the assembly with BACKUP_DESTINATION_ERROR: a missing index, a piece past the run's pieceCount, a byte total other than bytesWritten, and a digest other than the recorded one. The third and fourth catch the case where a later, smaller run reused the same name and left the earlier run's trailing pieces in place.

Use assembleFromDestination for a copy you took with backupTo. A chain of a full copy plus its change pieces needs restoreBackup, which the restore page covers.

Which route the bytes take

Sirannon takes one of two routes to your destination, and the report names the one it took in route.

The streamed route passes each piece to your destination as SQLite writes it, so the run needs no local disk at all. It works through a compiled SQLite extension that registers a virtual file system named sirannon, and Sirannon publishes that extension as one small package per platform. Installing the library fetches the package matching your os and cpu, because the library lists all six as optional dependencies pinned to its own version, so the install adds it for you.

The staged route writes one local file, then sends that file on in pieces. It needs free local disk the size of the backup, and localDiskRequired reads 'equal-to-backup' when Sirannon would take it.

Which route you get depends on the runtime. Node's own SQLite parses URI file names from version 23 upwards, and Sirannon names its virtual file system through a URI parameter, so the node driver streams on Node 23 and later and stages on Node 22. better-sqlite3 parses URI file names only where the operator sets SQLITE_USE_URI=1 before the module loads.

SQLITE_USE_URI=1 npx tsx check-route.ts

check-route.ts

import { Sirannon } from '@delali/sirannon-db'
import { betterSqlite3 } from '@delali/sirannon-db/driver/better-sqlite3'
 
const sirannon = new Sirannon({ driver: betterSqlite3() })
const db = await sirannon.open('orders', './data/orders.db')
 
const { streamedCopy, stagedCopy, localDiskRequired } = db.backupCapabilities()
 
console.log(JSON.stringify({ streamedCopy, stagedCopy, localDiskRequired }, null, 2))
 
await sirannon.shutdown()

The same file without that variable reports streamedCopy: false and localDiskRequired: 'equal-to-backup', which is the answer the backups page prints. The variable turns URI parsing on for every file name that process opens, so SQLite would then read a database path containing a question mark as a URI. Check the paths your application opens before you set it.

On a platform Sirannon publishes no binary for, compile the extension yourself and name it on the driver.

const driver = betterSqlite3({ vfsExtensionPath: '/opt/sirannon/sirannonvfs.so' })

The node driver takes the same option. A named path replaces the binary the install fetched, so naming one also pins a single build across a fleet.

The streamed route brings two constraints. It gives SQLite whole 512-byte blocks, so pieceBytes has to be a multiple of 512 and any other size fails with BACKUP_ERROR. Its fingerprint reads every piece back from your destination, because the run never holds the whole file, so a copy to remote storage does that read over the network. Set fingerprint: false where that read is more work than the check is worth.

Options

Option	Default	What it does
`destination`	required	Where the pieces go and where a restore reads them from.
`name`	`backup-{ISO timestamp}.db`	Name the pieces are stored under.
`chainId`	an identifier the run mints	The chain this copy begins.
`pieceBytes`	`16777216`	Bytes one whole piece holds. A streamed copy needs a multiple of 512.
`fingerprint`	`true`	Whether the run folds a SHA-256 over what it wrote.
`pagesPerStep`	`256`	Pages SQLite moves in one step.
`restartLimit`	`3`	Restarts the copy absorbs before it fails with `BACKUP_RESTARTED`.
`noProgressStepLimit`	`256`	Steps the copy may take without reaching a page it had not already copied.
`stallTimeoutMs`	`30000`	Milliseconds the copy may move no pages before it fails with `BACKUP_STALLED`.
`destinationTimeoutMs`	`600000`	Milliseconds one call to your destination may take. Zero leaves the calls unbounded.
`stagingDir`	the host temporary directory	Directory the staged route writes its local file in.
`onProgress`	none	Called at step resolution during the copy and once per piece during the transfer.

When a destination call hangs

A destination that never answers would hold a backup open for ever, because Sirannon counts a pending call as work in progress. destinationTimeoutMs bounds every individual call, and a call that passes it fails the run with BACKUP_DESTINATION_ERROR. Ten minutes is the default, which suits a large piece over a slow link, and zero removes the bound for a destination you trust to answer or throw on its own.

The same code covers a destination that refuses a call outright, and the message names the piece and the backup name it was stored under, so it points at one object in your storage. Over the server, BACKUP_DESTINATION_ERROR answers 502, which separates your storage failing from Sirannon failing.