Benchmarks¶

Performance claims without reproducible measurements are marketing. This page documents RockLake's benchmarking methodology, presents baseline results from controlled environments, explains how to reproduce benchmarks in your own infrastructure, and provides guidance on interpreting results. Every number on this page comes from automated benchmark suites that you can run yourself.

The goal of benchmarking is not to produce impressive numbers for a slide deck. The goal is to characterize the system's behavior so that operators can predict performance for their specific workload and deployment configuration. Accordingly, this page includes both best-case numbers (warm cache, co-located, S3 Express) and realistic numbers (mixed cache state, S3 Standard, typical production conditions).

Methodology¶

Framework¶

Benchmarks use the criterion framework for Rust, which provides:

• Statistical rigor: Each measurement is repeated until the confidence interval narrows to 5% or 100 iterations complete (whichever comes first)
• Warm-up iterations: The first N iterations are discarded to allow cache warming and JIT stabilization
• Outlier detection: Measurements that fall outside 3 standard deviations are flagged and excluded from percentile calculations
• Regression detection: Results are compared against previous runs to detect performance regressions

Test Environment¶

Baseline results on this page were produced on:

Catalog Setup¶

Each benchmark run creates a fresh catalog with deterministic content:

• 1 database, 5 schemas
• 20 tables per schema (100 total)
• 50 columns per table (5,000 total column rows)
• 100 data files per table (10,000 total file registration rows)
• 10 historical snapshots (some tables have been altered, creating version history)

This represents a moderately-sized analytics catalog — larger than a startup's first data warehouse, smaller than a Fortune 500 enterprise catalog.

Baseline Results: Point Operations¶

Point operations access a single entity by its key (or a small number of keys).

Cache-Hot Point Reads¶

After the benchmark warms up (first 100 iterations), all relevant SST blocks are in the block cache. These numbers represent steady-state performance for frequently-accessed data.

The hot key (system metadata) is served from application-level cache, bypassing SlateDB entirely. This makes it roughly 2–3x faster than regular cache-hit reads.

Cache-Cold Point Reads¶

These numbers represent the worst case: every read requires fetching an SST block from S3 Standard. This occurs on first access after startup or when accessing rarely-used catalog sections.

The variance in cold reads comes from S3's own latency variability, not from RockLake's processing. Note that each cold read also populates the cache, so subsequent reads of nearby keys (same SST block) will be cache-hot.

Baseline Results: Scan Operations¶

Scan operations read a range of keys with a common prefix (all columns of a table, all files for a table, all tables in a schema).

Cache-Hot Scans¶

Scan latency scales linearly with the number of rows returned. The cost is approximately 0.05–0.1ms per row for cache-hot data. This is dominated by MVCC filtering and response serialization, not by I/O.

Cache-Cold Scans¶

For large scans, the number of SST block fetches determines latency. Each block contains approximately 200–500 key-value pairs (depending on value size). A scan of 1,000 keys requires 3–5 block fetches, each adding ~50ms.

Baseline Results: Write Operations¶

All write operations go through the WAL, which requires one PUT to object storage regardless of batch size.

Observations on Write Performance¶

Batch size has minimal impact up to ~100 operations. The difference between registering 1 file (75ms) and 100 files (95ms) is only 20ms — because both result in one S3 PUT. The extra 20ms is the time to serialize a larger write batch.

At 1,000 operations per batch, serialization becomes visible. Constructing and serializing 2,002 key-value pairs adds ~50ms compared to a single-file registration. The S3 PUT itself also takes slightly longer because the payload is larger (~200KB vs. ~2KB).

Write throughput is limited by S3 PUT latency. Sequential writes (one transaction at a time) achieve 10–13 ops/s. This means approximately 1,000–1,300 files can be registered per second (at 100 files per batch). For most catalog workloads, this is more than sufficient.

S3 Express One Zone Results¶

For comparison, the same benchmarks on S3 Express One Zone storage:

S3 Express provides approximately 8–10x improvement on all storage-bound operations. Cache-hot operations see no improvement (they do not access storage).

Throughput Under Load¶

Sequential throughput (one client, one operation at a time):

Concurrent read throughput (multiple readers, no writer):

Read throughput scales well with concurrent readers because each reader has its own connection and accesses the immutable SST files independently. The slight per-reader throughput decrease at high concurrency is due to CPU scheduling and lock contention in the block cache.

Running Benchmarks¶

Prerequisites¶

# Rust toolchain (stable)
rustup update stable

# Criterion installed (comes with the workspace)
# No additional installation needed

Running the Standard Suite¶

cd crates/rocklake-catalog
cargo bench --bench catalog_bench

This runs the full benchmark suite and stores results in target/criterion/. Subsequent runs compare against previous results and report regressions.

Storage Configuration¶

# Local filesystem (fastest, for development — no network latency)
BENCHMARK_STORAGE=./bench-catalog cargo bench

# S3 Standard (realistic production latency)
AWS_REGION=us-east-1 BENCHMARK_STORAGE=s3://your-bench-bucket/catalog/ cargo bench

# S3 Express One Zone
AWS_REGION=us-east-1 BENCHMARK_STORAGE=s3express://your-express-bucket--use1-az4--x-s3/catalog/ cargo bench

# GCS
BENCHMARK_STORAGE=gs://your-bench-bucket/catalog/ cargo bench

Running Specific Benchmarks¶

# Only point read benchmarks
cargo bench --bench catalog_bench -- "point_read"

# Only write benchmarks
cargo bench --bench catalog_bench -- "write"

# Only scan benchmarks with specific size
cargo bench --bench catalog_bench -- "scan/1000"

Generating HTML Reports¶

Criterion generates HTML reports with interactive charts:

cargo bench --bench catalog_bench
# Reports at: target/criterion/report/index.html

Interpreting Results¶

Your Results Will Differ¶

The numbers on this page are from a controlled environment. Your results will differ based on:

Network distance. If DuckDB runs on a separate machine (not localhost), add the network round-trip time to every operation. For same-AZ, add 1–2ms. For cross-region, add 50–100ms.

S3 latency variability. S3 latency varies with time of day, region, and load. During peak hours, p99 latency can be 2–3x the off-peak values shown here.

Catalog size. The benchmark catalog has 100 tables. A catalog with 10,000 tables has more SST blocks, making cold-start scans proportionally slower. Cache-hot operations are unaffected by catalog size (as long as the working set fits in cache).

Cache warm-up time. The first 10–50 operations after startup will be slower (cold cache). If your workload has strong locality (same tables accessed repeatedly), the cache warms in seconds. If access is highly random, warm-up takes longer.

What Good Looks Like¶

For a DuckLake catalog serving an analytics workload:

• p50 read latency under 5ms indicates good cache utilization
• p99 read latency under 50ms indicates occasional cache misses (normal)
• Write latency under 150ms on S3 Standard is expected and acceptable
• Read throughput above 500 ops/s for a single reader indicates healthy operation

Red Flags¶

• p50 read latency consistently above 30ms suggests the cache is too small or the working set is too large
• Write latency above 300ms suggests S3 is experiencing unusual latency (check AWS status)
• Scan latency growing over time suggests GC has not run and scan amplification is increasing

Benchmark History¶

We maintain a history of benchmark results to track performance over time:

• Phase 2 Baseline — Initial benchmark results from the Phase 2 milestone
• v0.7 Performance Report — Current release benchmark data

Further Reading¶

• Latency Model — Understanding where time is spent
• Tuning — How to apply these insights to improve your deployment
• Operations: Monitoring — Tracking performance in production