Benchmarks

Sirannon benchmarks compare embedded SQLite performance against Postgres 17 across micro-operations, industry-standard workloads (YCSB, TPC-C), concurrency scaling, and Sirannon-specific features like CDC and connection pooling.

Two benchmark categories exist: local benchmarks for quick development feedback, and Docker-based benchmarks for fair, reproducible comparisons.

Prerequisites

• Node.js 22+ (with --expose-gc support)
• pnpm
• Docker and Docker Compose
• k6 (for end-to-end Docker benchmarks)
• Python 3 with matplotlib and pandas (for chart generation, optional)

Local Benchmarks

Local benchmarks use tinybench with --expose-gc enabled for accurate memory management between runs. Sirannon runs natively on the host; Postgres runs in a Docker container.

These benchmarks are useful for quick iteration during development, but the numbers reflect host-vs-container differences and vary across machines.

Starting Postgres

Most local benchmarks run Sirannon against a Postgres baseline. Start the Postgres container first:

docker compose -f benchmarks/docker-compose.yml up -d --wait

This launches Postgres 17 Alpine on port 5433 with tuned settings (256 MB shared buffers, synchronous_commit=off for reduced write latency, SSD-optimized planner costs). The default matched durability mode uses synchronous_commit=off for Postgres and PRAGMA synchronous=NORMAL for SQLite. Both sacrifice durability for write throughput, but at different layers. SQLite with synchronous=NORMAL in WAL mode syncs the WAL file at checkpoint boundaries rather than on every commit; a process crash is safe, but a power failure can lose transactions since the last checkpoint. Postgres with synchronous_commit=off reports a transaction as committed before the WAL is flushed to disk; a process crash is safe because the OS buffer cache remains intact, but an OS crash or power failure can lose up to ~600ms of recent transactions (3x the wal_writer_delay default of 200ms). Set BENCH_DURABILITY=full to enable maximum safety on both engines.

To stop and clean up:

docker compose -f benchmarks/docker-compose.yml down -v

Running Local Benchmarks

All commands run from packages/ts/.

Full suite:

pnpm bench

Runs every benchmark sequentially. Benchmarks that need Postgres are skipped if the container is not running.

Individual suites:

pnpm bench:micro       # point-select, bulk-insert, batch-update
pnpm bench:ycsb        # YCSB workload-a (50/50 mixed)
pnpm bench:oltp        # TPC-C lite
pnpm bench:scaling     # Concurrency scaling (event-loop + worker threads)
pnpm bench:pool        # Pool size sweep (5, 10, 20 connections)
pnpm bench:cdc         # CDC throughput and latency
pnpm bench:statistical # 4 key benchmarks x 10 runs with full statistics
pnpm bench:charts      # Generate SVG charts from CSV results

Local Benchmark Categories

Micro (requires Postgres) - Single-operation latency and throughput:

• point-select - Primary-key lookups
• bulk-insert - Batch row inserts
• batch-update - Batch row updates

YCSB (requires Postgres) - Yahoo! Cloud Serving Benchmark workloads:

• workload-a - 50% read, 50% update

OLTP (requires Postgres) - Online transaction processing:

• tpc-c-lite - Simplified TPC-C with 100K customers and 10K products, running new-order (45%), payment (43%), and order-status (12%) transactions

Scaling (requires Postgres) - Concurrency scaling from 1 to 64 clients:

• scaling/concurrency - Tests two deployment models: single event-loop (Sirannon serializes, Postgres overlaps via async pool) and worker thread pool (N threads with their own connections). Two workloads: read-only (WAL concurrent readers) and 50/50 mixed (single-writer contention).
• scaling/pool-sweep - Measures how Postgres connection pool size (5, 10, 20) affects throughput relative to Sirannon. Tests point-select and YCSB-A workloads at each pool size. Sirannon uses a fixed read pool of 4.

Sirannon (no Postgres needed) - Sirannon-specific features:

• cdc-latency - Change Data Capture event propagation latency
• connection-pool - Connection pool throughput and contention
• cold-start - Database open and first-query latency
• multi-tenant - Multi-database isolation throughput

Configuration

Override defaults with environment variables:

The default pool size of 10 connections follows standard Postgres sizing guidance, which recommends roughly 2x CPU cores plus disk spindles. Most production deployments use pools of 5-20 connections. Postgres performance degrades beyond this range because each connection is a separate OS process consuming memory and adding context-switch overhead. Tools like PgBouncer exist to multiplex thousands of application requests through a small connection pool.

The bulk-insert benchmark caps at 10K rows per iteration because each iteration inserts all rows one-by-one in a single transaction. At higher row counts, Postgres iterations take tens of seconds (one TCP round-trip per INSERT), producing too few samples for reliable measurement.

Example with custom settings:

BENCH_DATA_SIZES=1000,50000 BENCH_MEASURE_MS=20000 pnpm bench:micro

Docker Benchmarks (Fair, Reproducible)

Docker benchmarks run both Sirannon and Postgres inside containers with controlled CPU and memory limits. This produces fair comparisons because both databases operate under identical resource constraints.

Category 1: End-to-End (E2E) Application Benchmarks

Measures what a client experiences when an HTTP application uses Sirannon vs Postgres. Sirannon's architectural advantage (no network hop between app and DB) is part of the measurement.

Architecture:

• Sirannon app: Node.js + Sirannon in a single container (2 CPU, 2 GB)
• Postgres app: Node.js app container (1 CPU, 1 GB) + Postgres DB container (1 CPU, 1 GB)
• Total budget per setup: 2 CPU, 2 GB
• Load generator: k6 with constant-arrival-rate executor

k6 sends requests at a fixed rate regardless of response time, preventing coordinated omission. The result is a latency-throughput curve at each request rate.

Workloads:

• point-select.js - Single-row lookups by primary key
• mixed-readwrite.js - 80% read, 20% write (realistic web app)
• transaction.js - Multi-statement transactions

Running:

pnpm bench:docker:e2e

Configuration:

Category 2: Engine-Level Benchmarks

Measures raw query execution performance under identical constraints. No network latency in the measurement; each container runs tinybench internally.

Resource allocation:

• Sirannon engine: Sirannon + tinybench in a single container (2 CPU, 2 GB)
• Postgres engine: pg.Pool + tinybench in a client container (2 CPU, 2 GB) connected to a Postgres DB container (2 CPU, 2 GB)
• Sirannon uses 2 CPUs total. Postgres uses 4 CPUs total (2 client + 2 database). Sirannon runs as a single process with the database embedded in the application, so it needs one container. Postgres separates the database server from the client, requiring a container for each. This resource split reflects the architectural difference between embedded and client-server databases.

Both engines receive BENCH_DURABILITY via environment variable, so the matched vs full durability mode applies in Docker the same way it does locally.

Workloads (same as local benchmarks):

• Micro: point-select, bulk-insert, batch-update
• YCSB: workload A (50/50)
• OLTP: TPC-C lite

Running:

pnpm bench:docker:engine

Running both categories:

pnpm bench:docker

Configuration:

Statistical Analysis

Multi-run methodology

Set BENCH_RUNS to a value greater than 1 to enable multi-run analysis. Each run creates a fresh database, seeds it independently, and measures throughput. After all runs complete, the runner calculates:

• Welch t-test for statistical significance (unequal variance, N-1 degrees of freedom)
• 95% bootstrap confidence interval on the speedup ratio (10,000 resamples)
• IQR-based outlier detection on per-run ops/sec

The pnpm bench:statistical convenience script runs the 4 most representative benchmarks (point-select, batch-update, workload-a, tpc-c-lite) with BENCH_RUNS=10 at data sizes 1,000 and 10,000.

Reading statistical columns

The console table and CSV files include these columns when multi-run data is available:

• Sig: Significance stars. *** = p < 0.001, ** = p < 0.01, * = p < 0.05, n/s = not significant, - = single run.
• CI: 95% confidence interval on the speedup ratio, e.g. [4.2, 5.8] means the true speedup is between 4.2x and 5.8x with 95% confidence.
• Runs: Number of independent runs used for the calculation.

Speedup ratio as the primary metric

Absolute ops/sec varies by hardware, OS, thermal throttling, and background load. The speedup ratio (Sirannon ops/sec / Postgres ops/sec) cancels out most machine-specific factors and stays stable across different systems. When comparing results from different machines, focus on the ratio and CI, not raw throughput.

Results

Console output

Each run prints a comparison table to stdout with speedup and CI columns first:

Workload | N Rows | Speedup | CI | Sig | Sirannon ops/s | Postgres ops/s | P50 | P99 | CV | [Runs]

• Speedup - Sirannon ops/s divided by Postgres ops/s
• CI - 95% bootstrap confidence interval on speedup (multi-run only)
• Sig - Statistical significance (multi-run only)
• P50 / P99 - Sirannon median and 99th-percentile latency
• CV - Coefficient of variation; results marked [!] have CV > 10% and may be unreliable

JSON and CSV files

Raw results are written to benchmarks/results/ as timestamped JSON and CSV files. Each file contains system information, configuration, and per-workload results for both engines.

CSV files can be loaded directly into R, pandas, or any spreadsheet tool for custom analysis. When multi-run data is available, a separate *-per-run-*.csv file contains individual run data.

Charts

Generate SVG charts from CSV results:

pnpm bench:charts

This reads all CSV files from benchmarks/results/ and writes charts to benchmarks/results/charts/. Five chart types are generated:

• Speedup bar chart - Horizontal bars showing speedup ratio per workload, with CI error bars when available
• Scaling line chart - Ops/sec vs concurrency level for event-loop and worker-thread models
• Latency comparison - Grouped bars comparing P50 and P99 latencies between engines
• Feature bar chart - Ops/sec for Sirannon-only benchmarks (CDC, connection pool, cold start, multi-tenant)
• Per-run box plot - Distribution of ops/sec across runs, showing median, quartiles, and outliers for each workload

Requires Python 3 with matplotlib and pandas (pip install matplotlib pandas).

Reference Results

Results from a clean end-to-end run on Apple M3 Pro (11 cores, 18 GB RAM), Postgres 17.9 in Docker, matched durability mode. Statistical charts use 10 independent runs.

Speedup by Workload (10 runs, with 95% CI error bars)

Point-Select (primary key lookup, 1K and 10K rows):

Batch-Update (multi-row UPDATE in a transaction):

YCSB Workload-A (50/50 read/update, Zipfian distribution):

TPC-C Lite (mixed OLTP, 100K customers):

Speedup at Scale (1K to 1M rows)

Point-Select across data sizes:

YCSB-A across data sizes:

Batch-Update across data sizes:

Per-Run Distribution (10 runs)

Box plots showing the spread of ops/sec across independent runs. Tight boxes indicate stable measurements.

Point-Select:

TPC-C Lite:

Latency Comparison

Point-Select P50/P99:

Pool Size Sweep

Postgres pool size (5, 10, 20) vs Sirannon:

Sirannon Feature Benchmarks

CDC Throughput (sustained write rate with real-time event delivery):

Cold Start (database open + first query):

Connection Pool (internal pool throughput):

Interpreting results

• A speedup > 1 means Sirannon was faster than Postgres for that workload.
• CV > 10% indicates high variance in per-engine latency samples. For Sirannon's sub-10-microsecond operations (point-select, YCSB reads), high CV is expected: most samples complete in 2us, but occasional GC pauses, OS scheduler interrupts, and CPU frequency transitions inject multi-millisecond spikes into the tail. These outliers inflate the standard deviation far beyond the mean, producing CV values of 200-600% even though the median latency is stable. The speedup ratio and CI are the reliable metrics because both engines experience the same OS-level disruptions, and the ratio cancels out shared noise. If CV is high on operations that take milliseconds or more, close background apps and re-run with a longer BENCH_MEASURE_MS.
• Compare JSON files across runs to track regressions. System info is captured so you can account for hardware differences.
• Docker benchmarks on macOS use Docker Desktop with a Linux VM, adding overhead to both sides equally. Linux gives more accurate absolute numbers due to native Docker execution. Relative speedups should be consistent across platforms.

Concurrency and Scaling Trade-offs

The single-client benchmarks show Sirannon outperforming Postgres by 22x to 244x depending on the workload. These numbers are accurate, but they tell only half the story. The concurrency scaling benchmark (pnpm bench:scaling) reveals how these advantages shift as concurrent clients increase.

Why single-client performance is so high

Sirannon runs SQLite in-process. A query travels from your application code into SQLite's B-tree engine and back without leaving the process boundary. Postgres receives the same query over a TCP socket, parses it, plans it, executes it, serializes the result, and sends it back over the network. For a trivial point-select that takes microseconds to execute, the network round-trip dominates Postgres's total response time. Sirannon skips that round-trip entirely.

This explains the speedup hierarchy across workloads:

The pattern is clear: the less work each query does, the more network overhead dominates, and the larger Sirannon's advantage.

What happens under concurrency

SQLite uses a single-writer model. WAL mode allows any number of concurrent readers, but only one connection can write at a time. All other writers queue behind it. This is a fundamental architectural choice, not a bug or limitation to work around.

The scaling benchmark tests concurrency from 1 to 64 clients and reveals the consequences:

Event-loop model (single Node.js thread, async Postgres pool):

Sirannon's throughput stays flat regardless of how many concurrent clients are running. It can't go faster because the single event loop serializes all operations anyway, and SQLite's single-writer lock prevents write parallelism. Postgres, on the other hand, scales from 6,400 to 30,700 ops/sec on reads as its connection pool overlaps queries across the async event loop. Each additional Postgres connection does real parallel work on the server side.

The speedup ratio compresses from 66x down to 13x for reads, and from 22x down to 5x for mixed workloads. Sirannon still wins at every concurrency level tested, but the margin narrows as Postgres gains ground.

Worker-thread model (N threads, each with its own connection):

Worker threads unlock SQLite's read parallelism. WAL mode allows multiple threads to read simultaneously, and Sirannon's throughput jumps from 383K to 927K ops/sec at 4 threads for read-only workloads. Beyond 4 threads, the gains plateau because the hardware's memory bandwidth and cache contention become the bottleneck rather than SQLite's locking.

Mixed workloads tell a different story. Even with worker threads, write throughput stays flat at around 112-123K ops/sec because SQLite's write lock serializes all mutations regardless of how many threads are waiting. The extra threads don't help with writes; they only add contention overhead.

The crossover point

Postgres never catches Sirannon within the concurrency range tested (1 to 64 clients). At 64 concurrent mixed-workload clients, Sirannon still delivers 5x the throughput.

Postgres throughput grows linearly with concurrency while Sirannon's stays constant. Extrapolating the mixed-workload curve, Postgres would need roughly 300-400 concurrent database connections doing real parallel work to match Sirannon's throughput. In practice, production Postgres deployments rarely exceed 20-100 direct connections. Postgres documentation recommends keeping max_connections low, and tools like PgBouncer exist specifically to multiplex thousands of application requests through a small pool of database connections. The crossover point falls well outside normal production configurations.

Practical guidance

Sirannon outperforms Postgres across every workload and concurrency level tested, from 244x on batch updates down to 5x at 64 concurrent mixed clients. The concurrency range tested (1 to 64 clients) covers and exceeds what most production database connection pools use.

Web applications at standard scale see large, consistent gains. A backend handling hundreds or thousands of requests per day, with a connection pool of 10-50 concurrent database queries, falls squarely within the range where Sirannon leads by 5-22x on mixed workloads and 13-66x on reads. Sirannon also eliminates the operational overhead of running a separate database server. Connection pooling and configuration happen in-process rather than requiring external infrastructure to manage.

Multi-tenant architectures are a natural fit. Each tenant gets its own SQLite database, so write serialization applies per tenant rather than across the whole system. A tenant with 10 concurrent users gets the full single-client performance profile, and tenants don't contend with each other.

Read-heavy workloads at scale benefit from SQLite's WAL mode, which allows concurrent readers. The worker-thread benchmarks show Sirannon scaling from 383K to 927K read ops/sec at 4 threads. For horizontal read scaling beyond a single machine, WAL replication tools like Litestream can stream changes to read replicas via S3 for durability, or through faster transports like Redis Streams for sub-second replication lag.

Where Postgres is the better fit:

• Extremely write-heavy workloads with high concurrent writer counts on shared rows. SQLite serializes writes through a single-writer lock. The scaling benchmarks show Sirannon's write throughput stays flat as concurrency increases, while Postgres scales linearly with its row-level locking. For applications where hundreds of clients mutate the same tables simultaneously (high-frequency trading systems, real-time bidding platforms, shared counters with hundreds of concurrent writers), Postgres's ability to parallelize writes across connections is a genuine advantage. For most applications, SQLite's single writer processes mutations fast enough that the queue never becomes a bottleneck.
• Data exceeding a single machine's disk. SQLite operates on a single file on a single machine. If your dataset outgrows local storage, Postgres handles that natively with partitioning, tablespaces, and distributed extensions.

What these benchmarks do and don't measure

These benchmarks measure raw database throughput at concurrency levels that match real production deployments (1 to 64 concurrent clients). Sirannon is faster than Postgres across every tested scenario, with the advantage ranging from 5x under heavy concurrency to 244x for batch operations.

The benchmarks do not measure replication, failover, or distributed transactions. SQLite replication is available through external tools (Litestream for WAL streaming, cr-sqlite for CRDT-based multi-writer), while Postgres includes these capabilities as built-in features. The trade-off is between Sirannon's deployment simplicity and Postgres's integrated feature set; both approaches achieve the same end result through different means.

Tips

• Close resource-heavy applications before running benchmarks.

• Run the full suite at least twice; use the second run's numbers since the first run warms OS and Docker caches.

• For shorter feedback loops during development, reduce data sizes and measurement time:

  BENCH_DATA_SIZES=1000 BENCH_WARMUP_MS=1000 BENCH_MEASURE_MS=3000 pnpm bench:micro

• The --expose-gc flag is already included in the pnpm scripts, so manual garbage collection calls in benchmarks work correctly.

• Docker benchmarks build images on first run. Subsequent runs reuse cached layers unless source files change.

File Structure

benchmarks/
  config.ts, runner.ts, reporter.ts     # Shared infrastructure
  schemas.ts, engine.ts                 # Schema definitions, engine interface
  sirannon-engine.ts, postgres-engine.ts # Engine implementations
  run-all.ts                            # Local benchmark orchestrator
  run-statistical.ts                    # Multi-run statistical benchmark script
  micro/                                # Local micro benchmarks
  ycsb/                                 # Local YCSB benchmarks
  oltp/                                 # Local OLTP benchmarks
  sirannon/                             # Sirannon-only benchmarks (CDC, pool, etc.)
  scaling/                              # Concurrency and pool scaling benchmarks
  scripts/
    generate-charts.py                  # Chart generation from CSV results
  docker/                               # Docker-based fair benchmarks
    docker-compose.yml                  # Orchestrates all containers
    Dockerfile.sirannon-app             # Category 1: Sirannon HTTP server
    Dockerfile.postgres-app             # Category 1: Postgres-backed HTTP server
    Dockerfile.engine                   # Category 2: Engine benchmark runner
 
  e2e/                                  # Category 1 support files
    sirannon-app.ts                     # Sirannon HTTP server entrypoint
    postgres-app.ts                     # Postgres HTTP server entrypoint
    postgres-server.ts                  # Postgres HTTP server implementation
    seed.ts                             # Shared seeding utilities
 
  k6/                                   # Category 1 load scripts
    point-select.js                     # Single-row lookups
    mixed-readwrite.js                  # 80/20 read/write mix
    transaction.js                      # Multi-statement transactions
    helpers/zipfian.js                  # Zipfian distribution for k6
 
  engine/                               # Category 2 support files
    control-server.ts                   # Control API (setup, benchmark, cleanup)
    workloads.ts                        # Workload definitions
 
  run-e2e.ts                            # Orchestrate Category 1
  run-engine.ts                         # Orchestrate Category 2
  run-docker.ts                         # Run both categories
  results/                              # Output directory for JSON/CSV results and charts