Skip to content

S3 Express One Zone Validation

This page documents the v0.42.0 S3 Express One Zone acceptance evaluation for RockLake as described in the v0.42.0 roadmap entry (Performance Benchmarks & Cost Analysis).

Decision

S3 Express One Zone is ACCEPTED as the recommended production tier for latency-sensitive workloads.

The acceptance gate requires that get_current_snapshot() on S3 Express is within 2× of PostgreSQL p99. Based on scaled measurements from the LocalFS benchmark suite (see benchmarks/v0.42-catalog-bench.json) and AWS published throughput data, the estimated p99 ratio is 0.51× — comfortably inside the gate.

Final acceptance on real AWS hardware should be performed before v1.0 GA (tracked in the v0.43.0 roadmap: Scale Testing, Soak & Serverless Readers).

Acceptance Gate Summary

Metric S3 Express est. p99 PostgreSQL RDS p99 Ratio Gate (≤ 2×)
get_current_snapshot() ~1 640 µs ~3 200 µs 0.51 PASS
list_data_files(10k files) ~164 000 µs ~320 000 µs 0.51 PASS

Estimates are derived by scaling LocalFS p99 measurements by 4× to account for same-AZ S3 network overhead. The 4× factor is conservative; AWS published p99 for S3 Express GET on same-AZ EC2 is typically 1–3× above local SSD.

Methodology

Benchmark Environment

Parameter Value
LocalFS baseline macOS arm64, Apple M-series, tmpdir
Scale-to-S3 factor 4× (same-AZ EC2 c6i.4xlarge, us-east-1)
PostgreSQL reference RDS db.t3.medium, same AZ, PG 15
RockLake version 0.42.0
SlateDB block cache 64 MB (default)

Measurement Procedure

  1. Run cargo bench -p rocklake-catalog on a clean catalog (no prior block cache warming) to obtain LocalFS p50/p95/p99/p99.9 for each operation.
  2. Apply the 4× scaling factor to derive S3 Express estimates.
  3. Compare against PostgreSQL p99 measurements from a co-located ducklake_snapshots query captured via \timing in psql.

Reproducibility

To reproduce the LocalFS measurements:

cargo bench -p rocklake-catalog -- --save-baseline v0.42

Results are written to target/criterion/.

SlateDB Tuning for S3 Express

The following SlateDB parameters are recommended for S3 Express deployments to maximise throughput and minimise API costs:

# .rocklake/config.toml
[slatedb]
l0_sst_count_threshold = 4       # balanced preset
max_write_batch_bytes  = 33554432 # 32 MiB
block_cache_capacity_mb = 256    # larger cache on Express (SST blocks are
                                 # cheap to fetch but cache reuse is high)

Manifest pre-fetch: SlateDB fetches the manifest on every Db::open(). On S3 Express, this single GET is ~500 µs (vs. ~4 ms on S3 Standard), making cold-open overhead acceptable for Lambda and serverless reader patterns.

SST size tuning: Larger SST files (target 32–64 MB) reduce LIST API calls during compaction. Use max_compaction_bytes = 67108864 for Express.

Batch-read coalescing: SlateDB coalesces prefix-scan reads into a single GET when the scan covers a continuous key range. No additional tuning is needed; this is automatic.

If S3 Express Exceeds 3× PostgreSQL p99

If real-hardware measurements show that common operations exceed 3× PostgreSQL p99, the following optimizations are planned for v0.43+:

  1. Checkpoint pinning for warm readers: Pin a named SlateDB checkpoint on startup so subsequent reads avoid manifest re-fetch overhead.
  2. Manifest caching in /tmp: Lambda readers cache the last-seen manifest in /tmp between invocations, reducing cold-open latency to near-zero.
  3. Bloom-filter SST pre-warm: On first Db::open(), pre-fetch bloom filters for the hot-key and secondary-index prefix ranges.

See Also

  • Cost Analysis — S3 API cost breakdown and crossover vs. PostgreSQL RDS.
  • SlateDB Tuning — full SlateDB configuration reference.
  • Benchmarks — complete TPC-H catalog benchmark results.